nsaspy/prolog-rlm

Fork 0

[EPIC] Real constraint-solving RLM benchmark + durable context mounts #453

New issue

Closed

opened 2026-09-10 21:18:25 +00:00 by nsaspy · 1 comment

nsaspy commented

2026-09-10 21:18:25 +00:00

Owner

Goal

Turn the current live deep experiment into a real end-to-end reasoning benchmark that exercises planning, recursive RLM execution, constraint solving, and trusted verification, then immediately add a first-class durable context-mount abstraction without bloating the core model prompt.

This epic is intentionally ordered:

Real benchmark first — establish an honest measurement surface before changing context semantics.
Durable/persistent context mounts second — add reusable context lifetime/visibility semantics after the benchmark can measure regressions/improvements.

The architectural rule is:

core RLM prompt = minimal runtime contract
CLI/downstream harness prompts = allowed to be richer/task-specific
verification = trusted host logic, never model self-report
permanent context != permanently injected prompt text

Slice 1 — Replace token-echo depth test with a real constraint-solving benchmark

Problem

benchmark/rlm_live_deep_experiment.pl currently injects a fixed typed plan and asks nested model calls to return LIVE_DEEP_OK.

That is useful for provider/recursion plumbing, but it does not prove:

the root planner can choose a useful plan;
the model can decompose a hard problem;
RLM recursion improves or preserves solution quality;
the produced answer satisfies the task rather than merely containing an expected token;
Plan -> Execute -> Observe -> Verify semantics are correct for a real reasoning task.

The benchmark must stop treating a magic token as correctness.

Benchmark contract

Create a deterministic, hard, uniquely-solvable finite-domain constraint problem tailored to the runtime.

Target properties:

enough variables/domains that blind guessing is effectively impossible;
all-different constraints;
arithmetic constraints;
ordering constraints;
implications / biconditionals;
cardinality constraints;
cross-group dependencies;
locally plausible but globally invalid partial assignments;
exactly one valid global solution, proven independently by trusted Prolog.

Prefer a structured logic-grid/scheduling/resource-assignment CSP whose entire instance can be generated/represented declaratively and checked with clpfd or equivalent trusted Prolog constraints.

Required execution lanes

A. `core-minimal`

The authoritative architecture test.

The benchmark supplies only the actual task and normal runtime metadata/capabilities. It must not inject an exact plan or a giant benchmark-specific planner prompt.

Core should continue to provide only the minimal typed-plan runtime contract already owned by rlm_completion.

This lane tells us whether core prompting/runtime semantics are sufficient.

B. `harness-guided`

A comparison lane owned by benchmark/CLI/downstream code.

It may add task-specific planning guidance, decomposition advice, output-shape reminders, or other scaffolding.

This lane must not change trusted verification.

The delta between A and B is diagnostic:

minimal weak + guided strong -> improve core interface/prompt semantics carefully
both weak                  -> likely planning/model/runtime limitation
both solve + verify rejects -> output/handoff/verification mismatch
verify catches confident bad answer -> verifier working correctly

Do not silently move harness guidance into the core just to make the benchmark green.

Depth / RLM comparison

Run meaningful depth variants where supported, at minimum depth 0/1/2.

Depth must not simply add identical echo calls. The selected/generated plan must actually use recursion/decomposition for the deeper lanes when RLM is selected.

Record at least:

requested/actual recursion depth;
recursive call count;
model call count;
planner attempt count if observable;
prompt/completion/total tokens;
cost;
latency;
selected provider/model;
plan parsed/validated status;
verification status;
solution completeness;
violated constraint IDs on failure;
whether the solution is the unique satisfying assignment;
lane (core-minimal vs harness-guided).

Trusted verification

Correctness must be decided by Prolog, not by substring matching and not by the LLM saying "verified".

Implement a verifier that:

parses/normalizes the model's final structured assignment;
requires every variable exactly once;
rejects unknown/missing/duplicate values;
checks every CSP constraint independently;
checks the assignment equals a satisfying solution;
proves the fixture itself has exactly one solution (deterministic test/startup validation);
reports structured failure details.

Prefer reusing the existing Spec/Verify semantic split where practical:

Frozen Spec = desired solution/acceptance requirements
Plan        = proposed strategy
Execution   = produced assignment
Observation = normalized observed assignment/constraint evidence
Verify      = trusted acceptance decision

Do not create executable authority from model-generated Prolog.

TDD / deterministic tests

Before relying on real provider runs, add deterministic tests proving:

fixture has exactly one solution;
valid known solution passes;
single-field mutation fails;
incomplete assignment fails;
duplicate domain value fails;
malformed output fails safely;
verifier reports violated constraint IDs;
benchmark status is based on verification, not token presence;
core-minimal does not install benchmark-specific exact-plan instructions;
harness-guided guidance remains downstream-only;
depth metrics are internally consistent.

Existing provider/plumbing benchmark

Do not lose the current useful provider/recursion smoke coverage. Either:

retain the old echo-depth experiment under an explicitly named smoke/plumbing mode; or
extract its assertions into a smaller provider-recursion smoke benchmark/test.

The main deep-integration reasoning benchmark should become the real CSP test.

CLI / benchmark runner

Update benchmark/run.pl as needed so users can explicitly run the real benchmark and compare lanes. Keep existing deterministic/integration behavior backwards-compatible where reasonable.

Suggested modes may include:

deep-integration
constraint-integration
constraint-guided

Exact naming can follow existing conventions; avoid redundant aliases.

Slice 2 — Durable context mounts / lifetime + visibility

Begin immediately after the benchmark slice is landed/working.

Current behavior

rlm_context currently supports process-local context records. Caller-owned context_ref / context_handle values can survive multiple rlm_completion/4 calls because completion does not delete caller-owned handles.

However the built-in memory backend is explicitly non-persistent, so handles do not survive a process restart.

Design principle

Do not add only permanent(true).

Separate two independent semantic axes:

Lifetime

ephemeral
session
persistent

Model visibility

At minimum:

opaque
prompt

Default persistent context visibility must be opaque.

Persistent does not mean inject into every prompt forever.

The planner should receive bounded metadata / mount identity and use normal context operations when content is required.

Proposed host-facing abstraction

Conceptually:

context_mount(Name,
              Source,
              [ lifetime(persistent),
                scope(project(prolog_rlm)),
                visibility(opaque)
              ],
              Outcome).

Exact names are open to implementation review, but preserve the separation of lifetime, scope, visibility and source identity.

Persistence model

Persist the mount/source identity and policy, not a magical serialized live handle.

Conceptually:

persistent mount record
  name
  adapter/backend
  closed source_ref
  lifetime/scope/visibility policy
  provenance/fingerprint/version
        |
        v
process restart
        |
        v
rehydrate a fresh context_handle

Adapters remain trusted host boundaries. Model-generated data must not register executable adapter callbacks.

Scope

Support explicit scope so durable context does not become accidental global ambient state.

Initial scope vocabulary can be small, e.g.:

runtime
session(SessionId)
project(ProjectId)

Avoid user-global implicit injection as a default.

Required tests

caller-owned existing context remains reusable across completions;
ephemeral context is deleted after owned completion cleanup;
session context survives repeated calls in the process;
persistent mount metadata survives backend reopen/restart simulation;
persistent mount rehydrates to a fresh valid handle;
persistent opaque content is not copied into root prompt text automatically;
model sees only allowed bounded mount metadata;
explicit visibility(prompt) is opt-in and bounded;
deleting/unmounting invalidates future resolution safely;
scope mismatch denies resolution;
adapter executable callbacks remain trusted registry state and are never serialized as model-controlled data;
no capability/authority escalation occurs from persistence.

Prompt ownership / benchmark interpretation

Keep the library core intentionally minimal.

rlm_completion may describe:

typed plan format/runtime contract;
goal;
bounded context metadata;
root capabilities;
child capabilities;
active tool schemas;
concise recursive semantics.

It should not grow benchmark-specific chain-of-thought coaching, exact plans, or domain-specific CSP tutorials just to improve benchmark results.

CLI, benchmark harnesses, applications and downstream agents may add richer guidance deliberately and observably.

The real benchmark must report both minimal and guided behavior so prompt changes can be evaluated instead of guessed.

Implementation order

Re-read exact current main and existing live benchmark/test APIs.
Add deterministic CSP fixture + trusted verifier tests.
Replace/add real live benchmark lane with minimal core prompting.
Add harness-guided comparison lane.
Preserve provider/recursion smoke coverage separately.
Run focused deterministic tests and aggregate test gate.
Run live provider benchmark where credential is available; publish metrics/artifacts without pretending an unavailable credential is a correctness pass.
Only then begin durable context-mount implementation.
Add persistence/lifetime/visibility tests first.
Implement smallest durable mount backend/interface consistent with rlm_context adapters and existing artifact/storage abstractions.
Re-run benchmark to detect prompt/context regressions.

Acceptance gate

The epic is complete only when:

a real provider can be asked to solve a genuinely difficult, uniquely-solvable CSP through rlm_completion;
correctness is independently machine-verified by trusted Prolog;
depth 0/1/2 produce comparable structured metrics;
the benchmark distinguishes core-minimal vs downstream-guided prompting;
no exact injected plan is required for the authoritative benchmark lane;
existing provider/recursion plumbing coverage is preserved;
failures identify verifier/constraint details rather than merely missing a magic token;
persistent context has explicit lifetime, scope and visibility semantics;
persistent opaque context is reusable without automatic prompt injection;
persistent mounts survive backend reopen/restart through rehydration;
core prompt remains minimal and domain-neutral;
focused tests + aggregate deterministic gates pass.

Start implementation with Slice 1 immediately.

## Goal Turn the current live deep experiment into a real end-to-end reasoning benchmark that exercises **planning, recursive RLM execution, constraint solving, and trusted verification**, then immediately add a first-class durable context-mount abstraction without bloating the core model prompt. This epic is intentionally ordered: 1. **Real benchmark first** — establish an honest measurement surface before changing context semantics. 2. **Durable/persistent context mounts second** — add reusable context lifetime/visibility semantics after the benchmark can measure regressions/improvements. The architectural rule is: ```text core RLM prompt = minimal runtime contract CLI/downstream harness prompts = allowed to be richer/task-specific verification = trusted host logic, never model self-report permanent context != permanently injected prompt text ``` --- # Slice 1 — Replace token-echo depth test with a real constraint-solving benchmark ## Problem `benchmark/rlm_live_deep_experiment.pl` currently injects a fixed typed plan and asks nested model calls to return `LIVE_DEEP_OK`. That is useful for provider/recursion plumbing, but it does **not** prove: - the root planner can choose a useful plan; - the model can decompose a hard problem; - RLM recursion improves or preserves solution quality; - the produced answer satisfies the task rather than merely containing an expected token; - Plan -> Execute -> Observe -> Verify semantics are correct for a real reasoning task. The benchmark must stop treating a magic token as correctness. ## Benchmark contract Create a deterministic, hard, uniquely-solvable finite-domain constraint problem tailored to the runtime. Target properties: - enough variables/domains that blind guessing is effectively impossible; - all-different constraints; - arithmetic constraints; - ordering constraints; - implications / biconditionals; - cardinality constraints; - cross-group dependencies; - locally plausible but globally invalid partial assignments; - exactly one valid global solution, proven independently by trusted Prolog. Prefer a structured logic-grid/scheduling/resource-assignment CSP whose entire instance can be generated/represented declaratively and checked with `clpfd` or equivalent trusted Prolog constraints. ## Required execution lanes ### A. `core-minimal` The authoritative architecture test. The benchmark supplies only the actual task and normal runtime metadata/capabilities. It must **not** inject an exact plan or a giant benchmark-specific planner prompt. Core should continue to provide only the minimal typed-plan runtime contract already owned by `rlm_completion`. This lane tells us whether core prompting/runtime semantics are sufficient. ### B. `harness-guided` A comparison lane owned by benchmark/CLI/downstream code. It may add task-specific planning guidance, decomposition advice, output-shape reminders, or other scaffolding. This lane must not change trusted verification. The delta between A and B is diagnostic: ```text minimal weak + guided strong -> improve core interface/prompt semantics carefully both weak -> likely planning/model/runtime limitation both solve + verify rejects -> output/handoff/verification mismatch verify catches confident bad answer -> verifier working correctly ``` Do not silently move harness guidance into the core just to make the benchmark green. ## Depth / RLM comparison Run meaningful depth variants where supported, at minimum depth 0/1/2. Depth must not simply add identical echo calls. The selected/generated plan must actually use recursion/decomposition for the deeper lanes when RLM is selected. Record at least: - requested/actual recursion depth; - recursive call count; - model call count; - planner attempt count if observable; - prompt/completion/total tokens; - cost; - latency; - selected provider/model; - plan parsed/validated status; - verification status; - solution completeness; - violated constraint IDs on failure; - whether the solution is the unique satisfying assignment; - lane (`core-minimal` vs `harness-guided`). ## Trusted verification Correctness must be decided by Prolog, not by substring matching and not by the LLM saying "verified". Implement a verifier that: 1. parses/normalizes the model's final structured assignment; 2. requires every variable exactly once; 3. rejects unknown/missing/duplicate values; 4. checks every CSP constraint independently; 5. checks the assignment equals a satisfying solution; 6. proves the fixture itself has exactly one solution (deterministic test/startup validation); 7. reports structured failure details. Prefer reusing the existing Spec/Verify semantic split where practical: ```text Frozen Spec = desired solution/acceptance requirements Plan = proposed strategy Execution = produced assignment Observation = normalized observed assignment/constraint evidence Verify = trusted acceptance decision ``` Do not create executable authority from model-generated Prolog. ## TDD / deterministic tests Before relying on real provider runs, add deterministic tests proving: - fixture has exactly one solution; - valid known solution passes; - single-field mutation fails; - incomplete assignment fails; - duplicate domain value fails; - malformed output fails safely; - verifier reports violated constraint IDs; - benchmark status is based on verification, not token presence; - `core-minimal` does not install benchmark-specific exact-plan instructions; - `harness-guided` guidance remains downstream-only; - depth metrics are internally consistent. ## Existing provider/plumbing benchmark Do not lose the current useful provider/recursion smoke coverage. Either: - retain the old echo-depth experiment under an explicitly named smoke/plumbing mode; or - extract its assertions into a smaller provider-recursion smoke benchmark/test. The main `deep-integration` reasoning benchmark should become the real CSP test. ## CLI / benchmark runner Update `benchmark/run.pl` as needed so users can explicitly run the real benchmark and compare lanes. Keep existing deterministic/integration behavior backwards-compatible where reasonable. Suggested modes may include: ```text deep-integration constraint-integration constraint-guided ``` Exact naming can follow existing conventions; avoid redundant aliases. --- # Slice 2 — Durable context mounts / lifetime + visibility Begin immediately after the benchmark slice is landed/working. ## Current behavior `rlm_context` currently supports process-local context records. Caller-owned `context_ref` / `context_handle` values can survive multiple `rlm_completion/4` calls because completion does not delete caller-owned handles. However the built-in memory backend is explicitly non-persistent, so handles do not survive a process restart. ## Design principle Do **not** add only `permanent(true)`. Separate two independent semantic axes: ### Lifetime ```prolog ephemeral session persistent ``` ### Model visibility At minimum: ```prolog opaque prompt ``` Default persistent context visibility must be `opaque`. **Persistent does not mean inject into every prompt forever.** The planner should receive bounded metadata / mount identity and use normal context operations when content is required. ## Proposed host-facing abstraction Conceptually: ```prolog context_mount(Name, Source, [ lifetime(persistent), scope(project(prolog_rlm)), visibility(opaque) ], Outcome). ``` Exact names are open to implementation review, but preserve the separation of lifetime, scope, visibility and source identity. ## Persistence model Persist the mount/source identity and policy, **not a magical serialized live handle**. Conceptually: ```text persistent mount record name adapter/backend closed source_ref lifetime/scope/visibility policy provenance/fingerprint/version | v process restart | v rehydrate a fresh context_handle ``` Adapters remain trusted host boundaries. Model-generated data must not register executable adapter callbacks. ## Scope Support explicit scope so durable context does not become accidental global ambient state. Initial scope vocabulary can be small, e.g.: ```prolog runtime session(SessionId) project(ProjectId) ``` Avoid user-global implicit injection as a default. ## Required tests - caller-owned existing context remains reusable across completions; - ephemeral context is deleted after owned completion cleanup; - session context survives repeated calls in the process; - persistent mount metadata survives backend reopen/restart simulation; - persistent mount rehydrates to a fresh valid handle; - persistent opaque content is **not** copied into root prompt text automatically; - model sees only allowed bounded mount metadata; - explicit `visibility(prompt)` is opt-in and bounded; - deleting/unmounting invalidates future resolution safely; - scope mismatch denies resolution; - adapter executable callbacks remain trusted registry state and are never serialized as model-controlled data; - no capability/authority escalation occurs from persistence. --- # Prompt ownership / benchmark interpretation Keep the library core intentionally minimal. `rlm_completion` may describe: - typed plan format/runtime contract; - goal; - bounded context metadata; - root capabilities; - child capabilities; - active tool schemas; - concise recursive semantics. It should **not** grow benchmark-specific chain-of-thought coaching, exact plans, or domain-specific CSP tutorials just to improve benchmark results. CLI, benchmark harnesses, applications and downstream agents may add richer guidance deliberately and observably. The real benchmark must report both minimal and guided behavior so prompt changes can be evaluated instead of guessed. --- # Implementation order 1. Re-read exact current `main` and existing live benchmark/test APIs. 2. Add deterministic CSP fixture + trusted verifier tests. 3. Replace/add real live benchmark lane with minimal core prompting. 4. Add harness-guided comparison lane. 5. Preserve provider/recursion smoke coverage separately. 6. Run focused deterministic tests and aggregate test gate. 7. Run live provider benchmark where credential is available; publish metrics/artifacts without pretending an unavailable credential is a correctness pass. 8. Only then begin durable context-mount implementation. 9. Add persistence/lifetime/visibility tests first. 10. Implement smallest durable mount backend/interface consistent with `rlm_context` adapters and existing artifact/storage abstractions. 11. Re-run benchmark to detect prompt/context regressions. --- # Acceptance gate The epic is complete only when: - a real provider can be asked to solve a genuinely difficult, uniquely-solvable CSP through `rlm_completion`; - correctness is independently machine-verified by trusted Prolog; - depth 0/1/2 produce comparable structured metrics; - the benchmark distinguishes core-minimal vs downstream-guided prompting; - no exact injected plan is required for the authoritative benchmark lane; - existing provider/recursion plumbing coverage is preserved; - failures identify verifier/constraint details rather than merely missing a magic token; - persistent context has explicit lifetime, scope and visibility semantics; - persistent opaque context is reusable without automatic prompt injection; - persistent mounts survive backend reopen/restart through rehydration; - core prompt remains minimal and domain-neutral; - focused tests + aggregate deterministic gates pass. Start implementation with Slice 1 immediately.

~~nsaspy referenced this issue 2026-09-10 21:20:53 +00:00~~

[EPIC] Real constraint-solving RLM benchmark + durable context mounts #223

nsaspy commented

2026-09-10 21:21:41 +00:00

Author

Owner

Duplicate of #223 (pre-existing Forgejo mirror). Closing this accidental duplicate created by today's open-state sync; #223 stays canonical on Forgejo.

nsaspy closed this issue

2026-09-10 21:21:41 +00:00