EPIC: Context-quality evals, telemetry, and upstream-sync gates #7

New issue

Open

opened 2026-08-22 22:31:05 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-22 22:31:05 +00:00

(Migrated from github.com)

Parent: #1

Goal

Make context minimization and symbolic control measurable, regression-tested and safe to carry as an Agent Zero fork.

Eval dimensions

Track at minimum:

bootstrap/system prompt tokens
per-turn provider-visible tokens by context class
tool/schema tokens
retrieved memory/evidence tokens
task success
latency
repeated tool calls / loop count
context growth over long irrelevant history
worker fanout/depth
symbolic compiler fallback/error rate

Required adversarial cases

irrelevant-history growth should not reduce completion quality
relevant evidence buried among distractors remains recoverable
immediate user correction outranks stale context
cross-project memory contamination is rejected
unverified claims do not become durable facts
superseded facts disappear from active state but retain provenance
repeated identical tool failure trips circuit breaker
large tool output remains externalized
prompt injection in files/tool output cannot elevate authority

Baseline discipline

Every optimization must compare symbolic mode against the same Agent Zero model/task/tool set. Token reduction alone is not success if task completion regresses materially.

Upstream fork policy

Keep upstream Agent Zero mergeable where practical.
Prefer bundled plugin/extensions over invasive core edits.
Isolate unavoidable core patches and cover them with focused tests.
Before upstream sync, review affected symbolic hook points and run the context regression suite.
Never resolve upstream conflicts by silently dropping symbolic invariants.

RAGE loop

Review existing tests/telemetry and upstream delta -> analyze measurable hypothesis -> generate benchmark/regression -> execute baseline + symbolic runs -> inspect failures -> iterate until the hypothesis is actually supported.

Acceptance

Reproducible bootstrap/context benchmark exists.
Token ledger is emitted per call in symbolic mode.
Regression corpus covers context rot, selective forgetting, temporal updates and long-run tool loops.
CI catches accidental reintroduction of eager full-catalog prompting.
CI catches context budget overflow before provider invocation.
Upstream-sync checklist/test target is documented.
Symbolic mode has explicit quality and token thresholds for graduating from experimental to default.
Benchmark artifacts include enough metadata to reproduce model/config/context conditions.

Parent: #1 ## Goal Make context minimization and symbolic control measurable, regression-tested and safe to carry as an Agent Zero fork. ## Eval dimensions Track at minimum: - bootstrap/system prompt tokens - per-turn provider-visible tokens by context class - tool/schema tokens - retrieved memory/evidence tokens - task success - latency - repeated tool calls / loop count - context growth over long irrelevant history - worker fanout/depth - symbolic compiler fallback/error rate ## Required adversarial cases - irrelevant-history growth should not reduce completion quality - relevant evidence buried among distractors remains recoverable - immediate user correction outranks stale context - cross-project memory contamination is rejected - unverified claims do not become durable facts - superseded facts disappear from active state but retain provenance - repeated identical tool failure trips circuit breaker - large tool output remains externalized - prompt injection in files/tool output cannot elevate authority ## Baseline discipline Every optimization must compare symbolic mode against the same Agent Zero model/task/tool set. Token reduction alone is not success if task completion regresses materially. ## Upstream fork policy - Keep upstream Agent Zero mergeable where practical. - Prefer bundled plugin/extensions over invasive core edits. - Isolate unavoidable core patches and cover them with focused tests. - Before upstream sync, review affected symbolic hook points and run the context regression suite. - Never resolve upstream conflicts by silently dropping symbolic invariants. ## RAGE loop Review existing tests/telemetry and upstream delta -> analyze measurable hypothesis -> generate benchmark/regression -> execute baseline + symbolic runs -> inspect failures -> iterate until the hypothesis is actually supported. ## Acceptance - [ ] Reproducible bootstrap/context benchmark exists. - [ ] Token ledger is emitted per call in symbolic mode. - [ ] Regression corpus covers context rot, selective forgetting, temporal updates and long-run tool loops. - [ ] CI catches accidental reintroduction of eager full-catalog prompting. - [ ] CI catches context budget overflow before provider invocation. - [ ] Upstream-sync checklist/test target is documented. - [ ] Symbolic mode has explicit quality and token thresholds for graduating from experimental to default. - [ ] Benchmark artifacts include enough metadata to reproduce model/config/context conditions.