[P0] Deterministic PlUnit gate is false-green because nested runner replaces main #151

New issue

Closed

opened 2026-08-21 13:00:13 +00:00 by lost-rob0t · 2 comments

lost-rob0t commented

2026-08-21 13:00:13 +00:00

(Migrated from github.com)

Failure

The advertised deterministic PlUnit gate invoked by:

swipl -q -s test/run_tests.pl

does not currently execute the aggregate run_tests/0 suite.

test/run_tests.pl registers:

:- initialization(main, main).

but later consults test/run_tool_mcp_async_tests.pl, which registers:

:- initialization(focused_main, main).

SWI-Prolog's initialization(Goal, main) contract executes the last registered main goal. Therefore the consulted helper replaces the aggregate runner's entrypoint. focused_main runs only run_tool_mcp_async_cases and halts 0.

This exactly matches GitHub Actions logs: the Run deterministic PlUnit suite step prints only the canonical async cases and exits successfully; ordinary PlUnit units are never reached.

Concrete proof

Draft PR #149 added a deliberately failing conformance test for successful child-result propagation. Current production rlm_agent still only forwards kind:failed replies from children, yet the full deterministic Actions job stayed green. Strengthening the test assertions did not change that. The reason is the aggregate test main is never executed.

Impact

This is a repository-wide false-green condition for the deterministic PlUnit gate. CI success since this runner composition was introduced cannot by itself prove the full deterministic test corpus passed.

Other gates such as static load, Tree-sitter FFI, clean pack install, benchmarks, and live OpenRouter are separate and are not claimed broken by this issue.

Required fix

Keep the focused async runner usable standalone without letting it register/replace main when consumed by the aggregate runner. Prefer one of:

split reusable focused predicates into a helper module and keep the executable initialization(..., main) in a thin standalone entrypoint;
or otherwise guarantee test/run_tests.pl is the sole owner of initialization(..., main) in the aggregate program.

Do not merely reorder the directives: that leaves a fragile last-writer-wins entrypoint trap.

Acceptance

swipl -q -s test/run_tests.pl executes the full PlUnit corpus and exits nonzero when any included test fails;
run_tests([rlm_subagent_supervision]) on #149's test-only state is demonstrably red before the production success path is implemented;
the focused canonical async runner remains independently executable;
CI output provides positive evidence that the aggregate PlUnit suite ran (unit/summary/sentinel), not merely exit 0;
add a regression that prevents a consulted helper from silently replacing the aggregate main again.

Coordination

#149 is actively owned by another worker and already touches test/run_tests.pl; avoid parallel edits to that branch. The active worker can close this issue from #149 if it fixes the runner while implementing #144.

## Failure The advertised deterministic PlUnit gate invoked by: ```sh swipl -q -s test/run_tests.pl ``` does **not** currently execute the aggregate `run_tests/0` suite. `test/run_tests.pl` registers: ```prolog :- initialization(main, main). ``` but later consults `test/run_tool_mcp_async_tests.pl`, which registers: ```prolog :- initialization(focused_main, main). ``` SWI-Prolog's `initialization(Goal, main)` contract executes the **last** registered main goal. Therefore the consulted helper replaces the aggregate runner's entrypoint. `focused_main` runs only `run_tool_mcp_async_cases` and halts 0. This exactly matches GitHub Actions logs: the `Run deterministic PlUnit suite` step prints only the canonical async cases and exits successfully; ordinary PlUnit units are never reached. ## Concrete proof Draft PR #149 added a deliberately failing conformance test for successful child-result propagation. Current production `rlm_agent` still only forwards `kind:failed` replies from children, yet the full deterministic Actions job stayed green. Strengthening the test assertions did not change that. The reason is the aggregate test main is never executed. ## Impact This is a repository-wide false-green condition for the deterministic PlUnit gate. CI success since this runner composition was introduced cannot by itself prove the full deterministic test corpus passed. Other gates such as static load, Tree-sitter FFI, clean pack install, benchmarks, and live OpenRouter are separate and are not claimed broken by this issue. ## Required fix Keep the focused async runner usable standalone without letting it register/replace `main` when consumed by the aggregate runner. Prefer one of: - split reusable focused predicates into a helper module and keep the executable `initialization(..., main)` in a thin standalone entrypoint; - or otherwise guarantee `test/run_tests.pl` is the sole owner of `initialization(..., main)` in the aggregate program. Do not merely reorder the directives: that leaves a fragile last-writer-wins entrypoint trap. ## Acceptance - [ ] `swipl -q -s test/run_tests.pl` executes the full PlUnit corpus and exits nonzero when any included test fails; - [ ] `run_tests([rlm_subagent_supervision])` on #149's test-only state is demonstrably red before the production success path is implemented; - [ ] the focused canonical async runner remains independently executable; - [ ] CI output provides positive evidence that the aggregate PlUnit suite ran (unit/summary/sentinel), not merely exit 0; - [ ] add a regression that prevents a consulted helper from silently replacing the aggregate `main` again. ## Coordination #149 is actively owned by another worker and already touches `test/run_tests.pl`; avoid parallel edits to that branch. The active worker can close this issue from #149 if it fixes the runner while implementing #144.

lost-rob0t commented

2026-08-21 13:22:46 +00:00

(Migrated from github.com)

Trust-horizon audit: the false-green condition was introduced by merged #60 at commit 3db30d832f9b124d6b6e6aa3cbe2808914f4ae6e (2026-08-16). That same commit added test/run_tool_mcp_async_tests.pl with unconditional initialization(focused_main, main) and changed aggregate main(_) to call run_tool_mcp_async_cases before run_tests. Therefore deterministic PlUnit green results from #60 forward should not be treated as evidence that the full PlUnit corpus ran unless independently proven by another lane. Draft #152 is the isolated repair; it does not touch active #149.

Trust-horizon audit: the false-green condition was introduced by merged #60 at commit `3db30d832f9b124d6b6e6aa3cbe2808914f4ae6e` (2026-08-16). That same commit added `test/run_tool_mcp_async_tests.pl` with unconditional `initialization(focused_main, main)` and changed aggregate `main(_)` to call `run_tool_mcp_async_cases` before `run_tests`. Therefore deterministic PlUnit green results from #60 forward should not be treated as evidence that the full PlUnit corpus ran unless independently proven by another lane. Draft #152 is the isolated repair; it does not touch active #149.

lost-rob0t commented

2026-08-24 08:04:21 +00:00

(Migrated from github.com)

Completed on current main via the stronger runner integrated in #177. Exact current-main verification: runner-integrity 19/19; aggregate 74 suites, 762/762 passed, zero failures/timeouts/blocked/fixme. Stale implementation PR #152 was closed as superseded.