EPIC: Full-fidelity local I/O capture and debugging observability #59

New issue

Open

opened 2026-08-25 02:19:12 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-25 02:19:12 +00:00

(Migrated from github.com)

Parent: #1

Goal

Add an explicit Developer setting that can save the complete Agent Zero model/tool/runtime I/O stream locally for debugging difficult integration failures.

This is a diagnostic mode, not ordinary telemetry. It must be opt-in, obvious, local-first, bounded, and easy to delete.

User-facing setting

Add a Developer Settings control:

Save all I/O for debugging

Default: OFF.

When enabled, capture the complete supported debug envelope for each turn, including model request/response payloads, tool calls/results, transport decisions, Prolog-RLM compile requests/results, timing, token ledgers, provider error bodies, and relevant lifecycle events.

The UI must clearly warn that raw debug capture may contain sensitive data such as prompts, files, tool arguments/results, URLs, headers, environment-derived values, or credentials returned by external systems.

Storage

Default local location:

usr/debug/io/

Use structured append-only JSONL or equivalent machine-readable event files grouped by session/turn.

Each event should include enough correlation metadata to reconstruct a failing turn:

timestamp
session/chat/turn identifiers
event type
direction (in / out / internal)
subsystem (agent, litellm, openrouter, prolog_rlm, tool, api, websocket, etc.)
provider/model/mode when relevant
request/response IDs
payload
duration/status
token/context ledger
exception class/message/trace reference when relevant

Capture boundaries

Capture the actual payload as close as practical to each boundary rather than reconstructing it later.

Important boundaries:

final provider request immediately before LiteLLM/OpenRouter dispatch
raw provider response/error immediately after return
Agent Zero message-loop inputs/outputs
native tool schema selection
tool invocation args/results
Prolog-RLM context/compiler requests/results
transport fallback/retry decisions
API/WebSocket envelopes useful for reproducing UI/runtime failures

Security and privacy

Raw mode is intentionally dangerous and must not pretend otherwise.

Requirements:

OFF by default
local storage only by default
no automatic upload
no remote analytics sink
directory excluded from Git
restrictive file permissions where supported
visible warning before/when enabled
one-click/open-path indication for where files are written
clear/delete captured logs action
configurable retention/size bound
CI artifact upload must use a separate sanitized path, never blindly upload raw local captures

The normal application must continue redacting secrets in ordinary logs. Raw debug capture is a separate explicit facility.

Runtime behavior

Enabling/disabling should take effect without rebuilding the container.
Prefer no restart where technically practical.
Capture failures must never break normal agent operation.
Disk-full/write failures surface a warning and disable/fail-safe the recorder rather than crashing the agent.
Writes should not serialize the whole agent loop behind slow disk I/O.

Inspection UX

Developer Settings should also show:

enabled/disabled state
capture directory
current retained size
retention limit
last capture timestamp
clear captures button
optional download/export of a selected capture initiated by the user

A later viewer can provide structured filtering by subsystem/event/turn, but raw capture is the first requirement.

Tests

setting persists and defaults OFF
enable/disable behavior
provider request/response capture
provider error body capture
tool call/result capture
Prolog-RLM compile/result capture
retention/rotation
disk-write failure behavior
capture path remains under intended local root
ordinary logs remain redacted independently of raw mode
clear action deletes only debug-capture data

Acceptance

Developer Settings contains Save all I/O for debugging.
Default is OFF.
Raw model/provider I/O can be captured locally.
Raw tool I/O can be captured locally.
Prolog-RLM request/result data can be captured locally.
Transport retries/fallback decisions are recorded.
Capture files are structured and correlated by turn.
Storage is bounded and clearable.
Capture failures cannot crash the agent.
No capture is automatically uploaded.
CI diagnostics use sanitized artifacts and never treat this raw store as safe to publish.

Parent: #1 ## Goal Add an explicit Developer setting that can save the complete Agent Zero model/tool/runtime I/O stream locally for debugging difficult integration failures. This is a diagnostic mode, not ordinary telemetry. It must be opt-in, obvious, local-first, bounded, and easy to delete. ## User-facing setting Add a Developer Settings control: **Save all I/O for debugging** Default: OFF. When enabled, capture the complete supported debug envelope for each turn, including model request/response payloads, tool calls/results, transport decisions, Prolog-RLM compile requests/results, timing, token ledgers, provider error bodies, and relevant lifecycle events. The UI must clearly warn that raw debug capture may contain sensitive data such as prompts, files, tool arguments/results, URLs, headers, environment-derived values, or credentials returned by external systems. ## Storage Default local location: ```text usr/debug/io/ ``` Use structured append-only JSONL or equivalent machine-readable event files grouped by session/turn. Each event should include enough correlation metadata to reconstruct a failing turn: - timestamp - session/chat/turn identifiers - event type - direction (`in` / `out` / `internal`) - subsystem (`agent`, `litellm`, `openrouter`, `prolog_rlm`, `tool`, `api`, `websocket`, etc.) - provider/model/mode when relevant - request/response IDs - payload - duration/status - token/context ledger - exception class/message/trace reference when relevant ## Capture boundaries Capture the actual payload as close as practical to each boundary rather than reconstructing it later. Important boundaries: - final provider request immediately before LiteLLM/OpenRouter dispatch - raw provider response/error immediately after return - Agent Zero message-loop inputs/outputs - native tool schema selection - tool invocation args/results - Prolog-RLM context/compiler requests/results - transport fallback/retry decisions - API/WebSocket envelopes useful for reproducing UI/runtime failures ## Security and privacy Raw mode is intentionally dangerous and must not pretend otherwise. Requirements: - OFF by default - local storage only by default - no automatic upload - no remote analytics sink - directory excluded from Git - restrictive file permissions where supported - visible warning before/when enabled - one-click/open-path indication for where files are written - clear/delete captured logs action - configurable retention/size bound - CI artifact upload must use a separate sanitized path, never blindly upload raw local captures The normal application must continue redacting secrets in ordinary logs. Raw debug capture is a separate explicit facility. ## Runtime behavior - Enabling/disabling should take effect without rebuilding the container. - Prefer no restart where technically practical. - Capture failures must never break normal agent operation. - Disk-full/write failures surface a warning and disable/fail-safe the recorder rather than crashing the agent. - Writes should not serialize the whole agent loop behind slow disk I/O. ## Inspection UX Developer Settings should also show: - enabled/disabled state - capture directory - current retained size - retention limit - last capture timestamp - clear captures button - optional download/export of a selected capture initiated by the user A later viewer can provide structured filtering by subsystem/event/turn, but raw capture is the first requirement. ## Tests - setting persists and defaults OFF - enable/disable behavior - provider request/response capture - provider error body capture - tool call/result capture - Prolog-RLM compile/result capture - retention/rotation - disk-write failure behavior - capture path remains under intended local root - ordinary logs remain redacted independently of raw mode - clear action deletes only debug-capture data ## Acceptance - [ ] Developer Settings contains `Save all I/O for debugging`. - [ ] Default is OFF. - [ ] Raw model/provider I/O can be captured locally. - [ ] Raw tool I/O can be captured locally. - [ ] Prolog-RLM request/result data can be captured locally. - [ ] Transport retries/fallback decisions are recorded. - [ ] Capture files are structured and correlated by turn. - [ ] Storage is bounded and clearable. - [ ] Capture failures cannot crash the agent. - [ ] No capture is automatically uploaded. - [ ] CI diagnostics use sanitized artifacts and never treat this raw store as safe to publish.