nsaspy/prolog-rlm

Fork 0

[EPIC][expert] Review / critic expert: adversarial change analysis, regression discovery, and evidence-backed findings #412

New issue

Open

opened 2026-09-08 01:29:21 +00:00 by nsaspy · 3 comments

nsaspy commented

2026-09-08 01:29:21 +00:00

Owner

Parent: #376
Depends on: #377
Integrates with: #380, #382, #383, #384, #385

Goal

Add a reusable Review / Critic Expert that adversarially examines proposed/completed changes, plans and evidence for hidden regressions, unsupported assumptions, suspicious omissions and violated invariants before final acceptance.

The Review Expert is not the final verifier. It produces structured findings/evidence that the Planner/Repair/VERIFY path can consume.

Responsibilities

inspect changed artifacts/diffs and their dependency impact;
compare implementation state against Frozen Spec and project invariants;
search for unhandled callers/references/dependencies after changes;
detect suspiciously narrow verification or missing negative cases;
inspect Git/worktree evidence for unrelated or accidental changes;
identify security/authority/effect-boundary regressions from known rules;
identify stale evidence, unsupported claims and internal contradictions;
generate focused counterexample/evidence queries;
classify findings by severity/status/confidence and preserve provenance;
hand actionable findings to Repair Expert or missing-evidence requests to Retrieval/Project Knowledge experts.

Symbolic review first

Encode reusable deterministic review rules where possible, including:

changed exported symbol -> inspect known callers/references
changed schema           -> inspect producers + consumers
write affects file       -> require current post-write source generation
new effectful path       -> require capability/authority/effect evidence
removed obligation       -> hard finding against Frozen Spec
changed public API       -> inspect documented/observed dependents
new unknown branch       -> require negative/error-path evidence
unrelated diff           -> flag for explanation

Use Project Knowledge graph traversal and Git evidence rather than relying on prose summaries.

Findings contract

Normalize findings equivalent to:

finding id
rule/category
severity
status: confirmed | suspected | cleared | unknown
supporting evidence
counterevidence
affected requirement/artifact
recommended next evidence/repair goal

A suspected finding is not automatically a failure, but it cannot silently disappear. It must be cleared, accepted as a real violation, or remain explicit unknown/blocked evidence for final verification policy.

LLM fallback

This expert is a natural place for an optional adversarial model fallback on fuzzy semantic/code-review questions, but the symbolic review rules run first.

Model review output is candidate findings, not authoritative evidence. Every actionable claim should be grounded against current source/Git/test/runtime observations where possible.

Only explicit fallback calls are metered; Review Expert invocation itself is free.

Independence

The Write Expert should not review its own output as the only review path. Review should consume fresh external observations/diffs from Project Knowledge/Git sources.

VERIFY remains the final requirement acceptance boundary.

Acceptance

Deterministic fixture detects an unupdated dependent/reference with zero model calls.
Detects a new effectful path lacking required authority/effect evidence.
Detects unrelated changed file in a bounded diff fixture.
Project/Git evidence supporting each deterministic finding is preserved.
Suspected finding cannot be silently treated as cleared.
Cleared finding records the counterevidence that cleared it.
Actionable finding can become a Repair Expert subgoal.
Optional model fallback produces candidate findings that must be grounded/validated and is separately metered.
Model review prose alone cannot make final VERIFY succeed or fail.
Review cannot mutate Frozen Spec or widen capabilities.

Non-goals

No final acceptance authority.
No direct project mutation.
No requirement weakening.

Refs #376 #377 #380 #382 #383 #384 #385

Parent: #376 Depends on: #377 Integrates with: #380, #382, #383, #384, #385 ## Goal Add a reusable **Review / Critic Expert** that adversarially examines proposed/completed changes, plans and evidence for hidden regressions, unsupported assumptions, suspicious omissions and violated invariants before final acceptance. The Review Expert is not the final verifier. It produces structured findings/evidence that the Planner/Repair/VERIFY path can consume. ## Responsibilities - inspect changed artifacts/diffs and their dependency impact; - compare implementation state against Frozen Spec and project invariants; - search for unhandled callers/references/dependencies after changes; - detect suspiciously narrow verification or missing negative cases; - inspect Git/worktree evidence for unrelated or accidental changes; - identify security/authority/effect-boundary regressions from known rules; - identify stale evidence, unsupported claims and internal contradictions; - generate focused counterexample/evidence queries; - classify findings by severity/status/confidence and preserve provenance; - hand actionable findings to Repair Expert or missing-evidence requests to Retrieval/Project Knowledge experts. ## Symbolic review first Encode reusable deterministic review rules where possible, including: ```text changed exported symbol -> inspect known callers/references changed schema -> inspect producers + consumers write affects file -> require current post-write source generation new effectful path -> require capability/authority/effect evidence removed obligation -> hard finding against Frozen Spec changed public API -> inspect documented/observed dependents new unknown branch -> require negative/error-path evidence unrelated diff -> flag for explanation ``` Use Project Knowledge graph traversal and Git evidence rather than relying on prose summaries. ## Findings contract Normalize findings equivalent to: ```text finding id rule/category severity status: confirmed | suspected | cleared | unknown supporting evidence counterevidence affected requirement/artifact recommended next evidence/repair goal ``` A suspected finding is not automatically a failure, but it cannot silently disappear. It must be cleared, accepted as a real violation, or remain explicit unknown/blocked evidence for final verification policy. ## LLM fallback This expert is a natural place for an **optional adversarial model fallback** on fuzzy semantic/code-review questions, but the symbolic review rules run first. Model review output is candidate findings, not authoritative evidence. Every actionable claim should be grounded against current source/Git/test/runtime observations where possible. Only explicit fallback calls are metered; Review Expert invocation itself is free. ## Independence The Write Expert should not review its own output as the only review path. Review should consume fresh external observations/diffs from Project Knowledge/Git sources. VERIFY remains the final requirement acceptance boundary. ## Acceptance - [ ] Deterministic fixture detects an unupdated dependent/reference with zero model calls. - [ ] Detects a new effectful path lacking required authority/effect evidence. - [ ] Detects unrelated changed file in a bounded diff fixture. - [ ] Project/Git evidence supporting each deterministic finding is preserved. - [ ] Suspected finding cannot be silently treated as cleared. - [ ] Cleared finding records the counterevidence that cleared it. - [ ] Actionable finding can become a Repair Expert subgoal. - [ ] Optional model fallback produces candidate findings that must be grounded/validated and is separately metered. - [ ] Model review prose alone cannot make final VERIFY succeed or fail. - [ ] Review cannot mutate Frozen Spec or widen capabilities. ## Non-goals - No final acceptance authority. - No direct project mutation. - No requirement weakening. Refs #376 #377 #380 #382 #383 #384 #385

nsaspy commented

2026-09-08 01:46:51 +00:00

Author

Owner

MACHINE-SPIRIT #404A handoff — Review as counterexample evidence

EPOCH permits #386 findings to become negative/counterexample/evaluation evidence for candidate or admitted learned knowledge, but Review prose is not itself a promotion/retraction authority.

A material learning receipt should preserve finding ID/status, supporting/counterevidence, affected candidate/rule/method version and exact source/project generation. suspected findings cannot be silently treated as negative ground truth; cleared findings retain the evidence that cleared them. Promotion policy decides whether a finding blocks admission or triggers suspension.

Ref #404A EPOCH.

## MACHINE-SPIRIT #404A handoff — Review as counterexample evidence EPOCH permits #386 findings to become negative/counterexample/evaluation evidence for candidate or admitted learned knowledge, but Review prose is not itself a promotion/retraction authority. A material learning receipt should preserve finding ID/status, supporting/counterevidence, affected candidate/rule/method version and exact source/project generation. `suspected` findings cannot be silently treated as negative ground truth; `cleared` findings retain the evidence that cleared them. Promotion policy decides whether a finding blocks admission or triggers suspension. Ref #404A EPOCH.

nsaspy commented

2026-09-08 01:46:52 +00:00

Author

Owner

Machine Spirit #404B / CITADEL-EPOCH handoff

Review/Critic becomes one source for bounded adversarial counterexample campaigns during promotion, but preserves its existing non-authoritative role:

Review-generated findings/tests are candidate evidence until grounded by trusted semantics/observable evidence;
counterexample campaigns declare search profile, target assumptions, bounds and coverage; no counterexample found under a bound is not universal proof;
Review policy/learned review strategy may not weaken its own obligations, hide adverse cases, mark its own evidence independent, or self-promote;
candidate influence over Review inputs/routing is recorded as exposure/tamperability and prevents laundering the resulting finding as independent audit evidence.

Refs #404B CITADEL fixtures B17-B18, B20, B29, B34.

## Machine Spirit #404B / CITADEL-EPOCH handoff Review/Critic becomes one source for **bounded adversarial counterexample campaigns** during promotion, but preserves its existing non-authoritative role: - Review-generated findings/tests are candidate evidence until grounded by trusted semantics/observable evidence; - counterexample campaigns declare search profile, target assumptions, bounds and coverage; `no counterexample found` under a bound is not universal proof; - Review policy/learned review strategy may not weaken its own obligations, hide adverse cases, mark its own evidence independent, or self-promote; - candidate influence over Review inputs/routing is recorded as exposure/tamperability and prevents laundering the resulting finding as independent audit evidence. Refs #404B CITADEL fixtures B17-B18, B20, B29, B34.

nsaspy commented

2026-09-08 01:46:52 +00:00

Author

Owner

MACHINE-SPIRIT #404D / CLOCKWORK handoff

Review's adversarial-search output should use a typed counterexample campaign receipt rather than an unbounded prose claim. Record generator/search profile + version, seed/stream identity, finite domain or depth/work/case bounds, cases explored, stopping reason, found witnesses and minimized/shrunk witness where available.

Coverage classes must distinguish at least exhaustive, bounded, resource_limited, and unknown. Only a genuinely exhaustive finite/decidable search may emit exhaustive; no counterexample found under a bound never means universal correctness.

Review still cannot promote its own candidate or weaken its own evaluation policy. Refs #397 #404 #395.

## MACHINE-SPIRIT #404D / CLOCKWORK handoff Review's adversarial-search output should use a typed **counterexample campaign receipt** rather than an unbounded prose claim. Record generator/search profile + version, seed/stream identity, finite domain or depth/work/case bounds, cases explored, stopping reason, found witnesses and minimized/shrunk witness where available. Coverage classes must distinguish at least `exhaustive`, `bounded`, `resource_limited`, and `unknown`. Only a genuinely exhaustive finite/decidable search may emit `exhaustive`; `no counterexample found` under a bound never means universal correctness. Review still cannot promote its own candidate or weaken its own evaluation policy. Refs #397 #404 #395.

nsaspy referenced this issue

2026-09-08 01:48:17 +00:00

[ARADR][MACHINE-SPIRIT] 8 topics × 8 alternating research/design passes #387