Scheduler run contexts accumulate forever: one persisted chat per run, no eviction (and state_snapshot disagrees with get_task_by_run_id on previous_run_id) #119

New issue

Open

opened 2026-09-08 01:14:45 +00:00 by mara · 0 comments

mara commented

2026-09-08 01:14:45 +00:00

Collaborator

Summary

Commit 0b62cccb ("fix: isolate scheduler run contexts", PR #111, in main at bba5dcba) creates a fresh AgentContext with id=run_id for every task occurrence and persists it with save_tmp_chat (helpers/task_scheduler.py:998 and :1056, :1152, :1207). These run contexts are AgentContextType.TASK (not BACKGROUND), so save_tmp_chat (helpers/persist_chat.py:50-60) writes a chat file to usr/chats/<run_id>/ on every run. No code path ever removes them: helpers/task_scheduler.py contains no AgentContext.remove/remove_chat call, and the scheduler's own delete path (api/scheduler_task_delete.py:31-42) only removes the legacy task.uuid context and resets the current run context — previous run contexts are never cleaned.

Repro

Schedule a task every 20 minutes (e.g. task-steward-gate, cron */20).
Let it run for a few days.
ls usr/chats | wc -l grows by one directory per run occurrence (72/day for a 20-minute cadence). On the live instance, 73 chat folders exist with only 8 referenced by task run/context IDs — the rest accumulate as orphans.

Impact

Unbounded disk growth in usr/chats/ proportional to schedule frequency.
Each finished run context remains in AgentContext._contexts in-memory until process restart (nothing evicts it), holding agent history/log references.
Sidebar clutter: helpers/state_snapshot.py:312-320 only maps a run context to a task when ctx.id is the task's current_run_id or last_run_id. Contexts of previous runs fail that re-check (get_task_by_run_id at helpers/task_scheduler.py:739-752 does match previous_run_id, but state_snapshot.py:315-317 then nulls it because the id isn't current/last), so old run contexts fall through to legacy_task lookup (:318-320, requires last_run_id is None — false for any run task) and are emitted as plain chats in ctxs.

Location

helpers/task_scheduler.py:989-1012 (_new_run_context), :1056 and :1152/:1207 (_persist_chat calls), :947-982 (_finish_run — no cleanup).
helpers/state_snapshot.py:312-320 (previous-run contexts leak into the chats list).
Reviewed at main bba5dcba; introduced in 0b62cccb (2026-09-06).

Suggested fix

In _finish_run, after promoting the new last_run_id, evict the previous run: AgentContext.remove(task.previous_run_id) + persist_chat.remove_chat(task.previous_run_id) when it differs from current_run_id/last_run_id and is not a legacy task.context_id.
Alternatively, mark run contexts AgentContextType.BACKGROUND so save_tmp_chat skips them, and keep only the task-level previous_run_output (already bounded to 32 KiB) as the durable record; the WebUI task detail already renders previous_run_output from task serialization, not from the chat.
Whichever route: also make state_snapshot.py consistent with get_task_by_run_id (either include previous_run_id in the snapshot re-check or drop the redundant re-check entirely — today the two disagree about previous_run_id).

## Summary Commit `0b62cccb` ("fix: isolate scheduler run contexts", PR #111, in main at `bba5dcba`) creates a fresh `AgentContext` with `id=run_id` for every task occurrence and persists it with `save_tmp_chat` (`helpers/task_scheduler.py:998` and `:1056`, `:1152`, `:1207`). These run contexts are `AgentContextType.TASK` (not BACKGROUND), so `save_tmp_chat` (`helpers/persist_chat.py:50-60`) writes a chat file to `usr/chats/<run_id>/` on every run. No code path ever removes them: `helpers/task_scheduler.py` contains no `AgentContext.remove`/`remove_chat` call, and the scheduler's own delete path (`api/scheduler_task_delete.py:31-42`) only removes the legacy `task.uuid` context and resets the current run context — previous run contexts are never cleaned. ## Repro 1. Schedule a task every 20 minutes (e.g. `task-steward-gate`, cron `*/20`). 2. Let it run for a few days. 3. `ls usr/chats | wc -l` grows by one directory per run occurrence (72/day for a 20-minute cadence). On the live instance, 73 chat folders exist with only 8 referenced by task run/context IDs — the rest accumulate as orphans. ## Impact - Unbounded disk growth in `usr/chats/` proportional to schedule frequency. - Each finished run context remains in `AgentContext._contexts` in-memory until process restart (nothing evicts it), holding agent history/log references. - Sidebar clutter: `helpers/state_snapshot.py:312-320` only maps a run context to a task when `ctx.id` is the task's `current_run_id` or `last_run_id`. Contexts of *previous* runs fail that re-check (`get_task_by_run_id` at `helpers/task_scheduler.py:739-752` does match `previous_run_id`, but `state_snapshot.py:315-317` then nulls it because the id isn't current/last), so old run contexts fall through to `legacy_task` lookup (`:318-320`, requires `last_run_id is None` — false for any run task) and are emitted as plain chats in `ctxs`. ## Location - `helpers/task_scheduler.py:989-1012` (`_new_run_context`), `:1056` and `:1152`/`:1207` (`_persist_chat` calls), `:947-982` (`_finish_run` — no cleanup). - `helpers/state_snapshot.py:312-320` (previous-run contexts leak into the chats list). - Reviewed at main `bba5dcba`; introduced in `0b62cccb` (2026-09-06). ## Suggested fix - In `_finish_run`, after promoting the new `last_run_id`, evict the *previous* run: `AgentContext.remove(task.previous_run_id)` + `persist_chat.remove_chat(task.previous_run_id)` when it differs from `current_run_id`/`last_run_id` and is not a legacy `task.context_id`. - Alternatively, mark run contexts `AgentContextType.BACKGROUND` so `save_tmp_chat` skips them, and keep only the task-level `previous_run_output` (already bounded to 32 KiB) as the durable record; the WebUI task detail already renders `previous_run_output` from task serialization, not from the chat. - Whichever route: also make `state_snapshot.py` consistent with `get_task_by_run_id` (either include `previous_run_id` in the snapshot re-check or drop the redundant re-check entirely — today the two disagree about `previous_run_id`).

mara added the

bug

label

2026-09-08 01:14:45 +00:00