ADARD loop: build the recursive self-improvement research controller #15

New issue

Closed

opened 2026-07-20 00:21:30 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-07-20 00:21:30 +00:00

(Migrated from github.com)

Objective

Make the ADARD loop the primary experimental focus of starintel-auto-research because it has the strongest recursive self-improvement potential.

ADARD must convert the current durable Org-roam research workflow into an executable loop that can propose, test, compare, retain, and reuse improvements to its own research and development process.

Core loop

Assess the current pipeline, active problem, prior decisions, failures, and available evidence.
Design multiple materially different candidate improvements.
Analyze candidates through deterministic checks and independent reviewers.
Research missing evidence, alternatives, and falsifiers using bounded parallel branches.
Develop minimal prototypes or workflow changes on representative fixtures.
Evaluate each candidate against fixed metrics and baselines.
Promote, merge, prune, or retain candidates based on measured results.
Feed results back into memory, planning policy, skill selection, and future candidate generation.

RSI target

The loop should improve its own:

planning policy;
branch allocation and pruning;
context selection;
skill and model routing;
hypothesis quality;
evaluation strategy;
prototype workflow;
evidence quality;
cost, latency, and token efficiency;
ability to predict which changes will actually improve results.

Every claimed self-improvement must be tied to a reproducible run, a baseline, metrics, artifacts, and an evidence trail.

Initial deliverable

Create STAR-RESEARCH-PIPELINE-001 Adaptive ADARD Loop and Evaluation Harness covering only:

research-run ledger;
candidate and hypothesis schema;
exploration DAG;
branch lifecycle and budgets;
deterministic evaluation contract;
independent reviewer aggregation;
promotion gate;
feedback of evaluation results into future runs;
component ablation harness.

Semantic retrieval, full prototype execution, and learned planning policies should remain dependent designs unless required for the minimal loop.

Candidate lifecycle

QUEUED -> EXPLORING -> PROPOSING -> VIRTUAL_REVIEW -> PROTOTYPING -> EVALUATING -> PROMOTED | MERGED | PRUNED | BLOCKED

Multiple research candidates may exist simultaneously. The existing single promoted implementation-design slot remains unchanged.

Required metrics

At minimum, track:

task success;
evidence coverage and claim support;
unsupported-claim rate;
novelty versus prior candidates;
architecture fit;
implementation feasibility;
regression count;
token and monetary cost;
wall-clock latency;
branch yield;
prediction accuracy of candidate scores versus measured outcomes.

Promotion rules

A candidate may be promoted only when:

it beats the recorded baseline on the target metrics;
central claims have supporting evidence;
deterministic checks pass;
reviewer aggregation clears the threshold;
regressions and costs are recorded;
the run can be reproduced from stored inputs and configuration.

Unsupported central claims are a veto. Failed and pruned candidates remain searchable negative evidence.

First experiments

Compare single-path research against a 3-branch exploration DAG.
Compare lexical-only context retrieval against lexical plus Org-roam graph context.
Compare one reviewer against an independent reviewer pool with robust aggregation.
Compare unrestricted context loading against bounded branch-aware context packs.
Ablate planning, exploration, reasoning, memory, prototype workflow, and evaluation independently.

Acceptance criteria

One command creates a reproducible ADARD research run.
The run produces at least three distinct candidates.
Candidates retain parentage, evidence, falsifiers, scores, costs, and artifacts.
At least two candidates can be evaluated against the same fixed baseline.
Promotion is performed by code from recorded metrics, not informal judgment alone.
A promoted candidate can generate or update the next numbered design without bypassing the one-design implementation slot.
Failed candidates become retrievable negative evidence.
An ablation report identifies which loop components produced measurable improvement.
A later run can use earlier measured outcomes to alter branch allocation, retrieval, or candidate selection.

Constraint

Org files remain authoritative for human-maintained research and design. Ledgers, graphs, indexes, embeddings, evaluations, and run state are derived or append-only reproducible state. ADARD may propose and evaluate changes to itself, but promotion into the canonical workflow remains gated, attributable, reversible, and measurable.

## Objective Make the ADARD loop the primary experimental focus of `starintel-auto-research` because it has the strongest recursive self-improvement potential. ADARD must convert the current durable Org-roam research workflow into an executable loop that can propose, test, compare, retain, and reuse improvements to its own research and development process. ## Core loop 1. **Assess** the current pipeline, active problem, prior decisions, failures, and available evidence. 2. **Design** multiple materially different candidate improvements. 3. **Analyze** candidates through deterministic checks and independent reviewers. 4. **Research** missing evidence, alternatives, and falsifiers using bounded parallel branches. 5. **Develop** minimal prototypes or workflow changes on representative fixtures. 6. **Evaluate** each candidate against fixed metrics and baselines. 7. **Promote, merge, prune, or retain** candidates based on measured results. 8. **Feed results back** into memory, planning policy, skill selection, and future candidate generation. ## RSI target The loop should improve its own: - planning policy; - branch allocation and pruning; - context selection; - skill and model routing; - hypothesis quality; - evaluation strategy; - prototype workflow; - evidence quality; - cost, latency, and token efficiency; - ability to predict which changes will actually improve results. Every claimed self-improvement must be tied to a reproducible run, a baseline, metrics, artifacts, and an evidence trail. ## Initial deliverable Create `STAR-RESEARCH-PIPELINE-001 Adaptive ADARD Loop and Evaluation Harness` covering only: - research-run ledger; - candidate and hypothesis schema; - exploration DAG; - branch lifecycle and budgets; - deterministic evaluation contract; - independent reviewer aggregation; - promotion gate; - feedback of evaluation results into future runs; - component ablation harness. Semantic retrieval, full prototype execution, and learned planning policies should remain dependent designs unless required for the minimal loop. ## Candidate lifecycle `QUEUED -> EXPLORING -> PROPOSING -> VIRTUAL_REVIEW -> PROTOTYPING -> EVALUATING -> PROMOTED | MERGED | PRUNED | BLOCKED` Multiple research candidates may exist simultaneously. The existing single promoted implementation-design slot remains unchanged. ## Required metrics At minimum, track: - task success; - evidence coverage and claim support; - unsupported-claim rate; - novelty versus prior candidates; - architecture fit; - implementation feasibility; - regression count; - token and monetary cost; - wall-clock latency; - branch yield; - prediction accuracy of candidate scores versus measured outcomes. ## Promotion rules A candidate may be promoted only when: - it beats the recorded baseline on the target metrics; - central claims have supporting evidence; - deterministic checks pass; - reviewer aggregation clears the threshold; - regressions and costs are recorded; - the run can be reproduced from stored inputs and configuration. Unsupported central claims are a veto. Failed and pruned candidates remain searchable negative evidence. ## First experiments 1. Compare single-path research against a 3-branch exploration DAG. 2. Compare lexical-only context retrieval against lexical plus Org-roam graph context. 3. Compare one reviewer against an independent reviewer pool with robust aggregation. 4. Compare unrestricted context loading against bounded branch-aware context packs. 5. Ablate planning, exploration, reasoning, memory, prototype workflow, and evaluation independently. ## Acceptance criteria - [ ] One command creates a reproducible ADARD research run. - [ ] The run produces at least three distinct candidates. - [ ] Candidates retain parentage, evidence, falsifiers, scores, costs, and artifacts. - [ ] At least two candidates can be evaluated against the same fixed baseline. - [ ] Promotion is performed by code from recorded metrics, not informal judgment alone. - [ ] A promoted candidate can generate or update the next numbered design without bypassing the one-design implementation slot. - [ ] Failed candidates become retrievable negative evidence. - [ ] An ablation report identifies which loop components produced measurable improvement. - [ ] A later run can use earlier measured outcomes to alter branch allocation, retrieval, or candidate selection. ## Constraint Org files remain authoritative for human-maintained research and design. Ledgers, graphs, indexes, embeddings, evaluations, and run state are derived or append-only reproducible state. ADARD may propose and evaluate changes to itself, but promotion into the canonical workflow remains gated, attributable, reversible, and measurable.