nsaspy/prolog-rlm

Fork 0

[P1] Add backend-neutral durable scoped state and policy APIs #75

New issue

Open

opened 2026-08-17 07:40:28 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-17 07:40:28 +00:00

(Migrated from github.com)

Parent: #74

Goal

Add the generic prolog-rlm core substrate for durable scoped trusted state: host/user/project/session scope identity, backend-neutral storage, validated values/rules, provenance, revocation/supersession, and deterministic resolution of ordinary settings.

This is the storage/policy substrate required by persistent project authorization and by downstream Prolog-native project configuration. It is not itself a coding-agent configuration product and it is not an arbitrary model-writable Prolog database.

Problem / motivation

Current core state deliberately serves different purposes:

rlm_authority holds mutable runtime/session authority and pending operations;
rlm_artifact holds immutable/versioned durable task knowledge and fresh-root handoff;
rlm_context exposes bounded opaque epistemic context;
rlm_graph persists resumable graph checkpoints/events;
MCP server/profile facts describe trusted host lifecycle configuration.

No current API means:

“This trusted operator/host selected this setting or policy value for this scope; preserve it across restart and resolve it deterministically later.”

That semantic class deserves its own closed contract rather than being aliased to artifacts, graph state, MCP facts, or authority history.

Architectural boundary

Core owns:

scope records and validation;
canonical scope identity;
project identity contract;
state/policy record schema;
trusted mutation API;
read/list/query/revoke/supersede semantics;
ordinary-setting precedence;
provenance/trust classification;
backend-neutral store lifecycle;
persistent restart semantics;
safe malformed/stale-state behavior.

Downstream packages own:

product-specific setting names;
user-facing config UX;
coding-agent-specific project conventions;
concrete policy-pack filename/directory conventions;
button wording such as “Always allow.”

This issue must not create a .claude clone or a giant fact/7 bucket.

Required scope model

Define closed scope forms equivalent in semantics to:

host
user(UserIdentity)
project(ProjectIdentity)
session(SessionIdentity)

Exact terms may follow repository conventions.

Project identity contract

Project identity is security-relevant because persistent authorization will depend on it. Do not key durable state by raw cwd.

The design/implementation must explicitly define behavior for:

canonical repository root when Git is available;
path canonicalization and symlinks;
filesystem relocation;
Git worktrees;
multiple clones;
non-Git projects;
stale/missing roots;
identity schema/version evolution.

Do not overbuild distributed project identity in the first slice. The required invariant is that project identity is a first-class structured concept distinct from current filesystem location.

The implementation should document whether relocation/worktrees/clones preserve or intentionally create distinct identities and provide an explicit migration/rebinding path if identities can become stale. Ambiguity at a security boundary must fail closed rather than silently choosing another project.

State record semantics

Define a closed, ground, schema-validated record capable of representing ordinary trusted settings and policy records without executing them.

It should carry the repository-convention equivalent of:

state/rule identity;
scope;
key/policy class;
normalized value or declarative rule data;
schema/version;
origin/trust class;
source/actor metadata where representable;
creation/update metadata where consistent with current conventions;
current/superseded/revoked status or equivalent version lineage.

Do not store arbitrary executable closures/callables as values.

A later value may supersede an older one while preserving enough provenance/history to explain what changed. Revocation must be a first-class semantic operation, not “write magic false into the same key.”

Following #64, active executable/effective policy state and retained audit/history must remain separable. History must not retain trusted continuation callables or become the live registry merely because append-only storage is convenient.

Public API direction

Follow existing structured-outcome naming/style. Exact predicate names are intentionally left to implementation, but the public surface should provide semantics equivalent to:

scoped_state_store_open(+BackendSpec, -Outcome).
scoped_state_store_close(+Store, -Outcome).

scope_resolve(+ScopeSpec, +Options, -Outcome).
project_identity(+ProjectHint, +Options, -Outcome).

scoped_state_get(+Store, +Scope, +Key, -Outcome).
scoped_state_list(+Store, +Scope, +Options, -Outcome).
scoped_state_query(+Store, +Query, -Outcome).

scoped_state_set(+Store, +TrustedMutation, -Outcome).
scoped_state_revoke(+Store, +StateRefOrRuleId, +Reason, -Outcome).

scoped_state_effective(+Store, +Scopes, +Key, +Options, -Outcome).
scoped_state_status(+Store, +Ref, -Outcome).
scoped_state_reload(+Store, -Outcome).

Names/arity may differ. The important contract is explicit store/scope identity, trusted mutation, inspectability, revocation, effective resolution, and backend independence.

Trusted mutation boundary

Mutation predicates are trusted host/library APIs, analogous to authority setters. They must not be registered as unrestricted model tools.

A model can produce a request or candidate setting/policy as ordinary data. Converting that request into durable trusted state requires a mediated host/operator action.

The API should make origin/trust explicit enough that loading arbitrary project/model data cannot masquerade as a host/operator mutation.

Backend contract

Provide at least:

deterministic in-memory backend for tests;
a persistent backend that survives process close/reopen/restart.

The public semantics must not depend on SWI library(persistency) even if the first persistent implementation reuses that mechanism, as rlm_artifact and rlm_graph currently do.

Backend requirements:

ground canonical data only;
deterministic open/close/reload behavior;
structured errors for malformed/corrupt/unsupported records;
schema/version checks before state becomes effective;
no implicit execution during loading;
atomic enough mutation semantics that a partially written revoke/update cannot accidentally leave a broader security policy active.

If a future remote/latency-bearing backend is added, follow #54: canonical execute semantics -> async Future -> synchronous facade awaiting the same Future. Do not add fake async variants to pure/in-memory operations merely for visual symmetry.

Precedence semantics

This issue owns deterministic precedence for ordinary non-security settings.

Define and test a documented scope overlay order. A likely shape is narrower scope overriding broader scope where eligible, but implementation must decide exact host/user/project/session ordering and how “unset”, revoked, invalid, and explicit values behave.

Do not reuse ordinary overlay logic as the security-authority algorithm.

Authority-affecting persisted rules are resolved by #76 through the #53 authority boundary and remain capped by current hard/host policy. This substrate stores and queries them but does not decide that they authorize execution.

Security invariants

Model output cannot call an unrestricted durable mutation primitive.
Persisted records are closed validated data, not arbitrary executable Prolog.
No arbitrary loaded term is passed to unrestricted call/1.
Project files do not become trusted state merely because they were discovered.
Loading/reloading state cannot grant tool capabilities.
Loading/reloading state cannot set dangerous or make allow_session persistent.
Malformed, unknown-schema, ambiguous-project, or untrusted authority-affecting state fails closed.
Persistent values cannot bypass schemas, path confinement, process/network policy, budgets, cancellation, capabilities, or effect accounting.
Secrets are represented by references to trusted external configuration where needed; ordinary scoped records must not become a plaintext secret store by default. Preserve the separation hardened by #52/#72.
Audit/history does not retain executable continuations.

Interaction with existing modules/issues

#16 / `rlm_artifact`

Reuse persistence/versioning lessons where useful, not artifact semantics. Artifacts are model/task knowledge and can be published by reasoning workflows; that must never imply permission mutation.

`rlm_graph` / agent state

Graph checkpoints and agent runtime state remain execution state. They may carry references to scoped state but are not the durable settings database.

#53 / `rlm_authority`

This issue does not persist authority modes and does not implement authorization decisions. #76 composes scoped policy with #53.

#52 / `rlm_mcp_policy`

MCP profile/config-reference facts are hard trusted lifecycle configuration. They remain separate from user/project preferences and remembered permissions.

#68-#71

#70 may resolve effective operator/project policy as a context source, but its rlm_artifact snapshots remain epistemic resolved-context records. Preserve provenance classes rather than flattening the stores.

#48/#49/#50

The core abstraction must be generic enough for external tool libraries and downstream agents without importing their concrete catalogs or UX.

Acceptance criteria

Closed scope terms exist for host/user/project/session or documented repository-equivalent forms.
Project identity is structured and not equal to raw cwd.
Deterministic tests cover symlink/canonical-root behavior and at least explicit worktree/multiple-clone/non-Git identity decisions.
A memory backend and persistent backend expose the same record/read/list/revoke/effective semantics.
A persistent project setting survives close/reopen in a fresh Prolog process.
Project A and project B can hold the same key with different values without leakage.
Ordinary setting precedence across configured scopes is deterministic and documented.
Trusted mutation records provenance/origin and rejects non-ground/malformed values.
Revocation makes a record ineligible immediately and after restart.
Superseded/revoked records remain inspectable without remaining active.
Unknown schema/version or malformed persistent records fail closed and never become effective authority policy.
A model-produced term cannot directly mutate trusted durable state through any model-facing plan/tool surface.
Loading an arbitrary project file does not mutate the scoped store.
State reload does not change the current authority tier and never converts allow_session to persistent state.
No generic fact/7 API collapses artifacts, repository context, MCP declarations, and trusted operator state into one semantic class.
Public exports and documentation clearly identify trusted mutation APIs versus read/inspection APIs.
Tests prove active state and audit/history are separable and no executable callback/continuation is retained for history.

Non-goals

No persistent project authorization matching/decision logic in this issue; see #76.
No concrete project-local Prolog pack/file convention; see #77.
No coding-agent-specific preference schema.
No UI/TUI.
No arbitrary Prolog code execution from persisted values.
No general secret-management service.
No fake async API for pure operations.
Do not replace rlm_artifact, rlm_context, or rlm_graph storage with this substrate.

Dependencies / references

Parent #74
#16 durable artifacts
#48 external tool-library boundary
#52 completed MCP declarative/config policy
#53 host-controlled authority
#54 sync/async contract
#64 active-vs-terminal authority retention lesson
#68-#71 context/workflow pipeline
research/RLM-RESEARCH-003-typed-symbolic-execution.org
research/RLM-RESEARCH-005-swi-agent-runtime.org
research/RLM-RESEARCH-007-langchain-langgraph-port.org
research/RLM-RESEARCH-009-durable-artifact-context.org

Blocks #76 and #77.

Parent: #74 ## Goal Add the generic `prolog-rlm` core substrate for **durable scoped trusted state**: host/user/project/session scope identity, backend-neutral storage, validated values/rules, provenance, revocation/supersession, and deterministic resolution of ordinary settings. This is the storage/policy substrate required by persistent project authorization and by downstream Prolog-native project configuration. It is **not** itself a coding-agent configuration product and it is not an arbitrary model-writable Prolog database. ## Problem / motivation Current core state deliberately serves different purposes: - `rlm_authority` holds mutable runtime/session authority and pending operations; - `rlm_artifact` holds immutable/versioned durable task knowledge and fresh-root handoff; - `rlm_context` exposes bounded opaque epistemic context; - `rlm_graph` persists resumable graph checkpoints/events; - MCP server/profile facts describe trusted host lifecycle configuration. No current API means: > “This trusted operator/host selected this setting or policy value for this scope; preserve it across restart and resolve it deterministically later.” That semantic class deserves its own closed contract rather than being aliased to artifacts, graph state, MCP facts, or authority history. ## Architectural boundary Core owns: - scope records and validation; - canonical scope identity; - project identity contract; - state/policy record schema; - trusted mutation API; - read/list/query/revoke/supersede semantics; - ordinary-setting precedence; - provenance/trust classification; - backend-neutral store lifecycle; - persistent restart semantics; - safe malformed/stale-state behavior. Downstream packages own: - product-specific setting names; - user-facing config UX; - coding-agent-specific project conventions; - concrete policy-pack filename/directory conventions; - button wording such as “Always allow.” This issue must not create a `.claude` clone or a giant `fact/7` bucket. ## Required scope model Define closed scope forms equivalent in semantics to: ```text host user(UserIdentity) project(ProjectIdentity) session(SessionIdentity) ``` Exact terms may follow repository conventions. ### Project identity contract Project identity is security-relevant because persistent authorization will depend on it. Do not key durable state by raw `cwd`. The design/implementation must explicitly define behavior for: - canonical repository root when Git is available; - path canonicalization and symlinks; - filesystem relocation; - Git worktrees; - multiple clones; - non-Git projects; - stale/missing roots; - identity schema/version evolution. Do not overbuild distributed project identity in the first slice. The required invariant is that **project identity is a first-class structured concept distinct from current filesystem location**. The implementation should document whether relocation/worktrees/clones preserve or intentionally create distinct identities and provide an explicit migration/rebinding path if identities can become stale. Ambiguity at a security boundary must fail closed rather than silently choosing another project. ## State record semantics Define a closed, ground, schema-validated record capable of representing ordinary trusted settings and policy records without executing them. It should carry the repository-convention equivalent of: - state/rule identity; - scope; - key/policy class; - normalized value or declarative rule data; - schema/version; - origin/trust class; - source/actor metadata where representable; - creation/update metadata where consistent with current conventions; - current/superseded/revoked status or equivalent version lineage. Do not store arbitrary executable closures/callables as values. A later value may supersede an older one while preserving enough provenance/history to explain what changed. Revocation must be a first-class semantic operation, not “write magic false into the same key.” Following #64, active executable/effective policy state and retained audit/history must remain separable. History must not retain trusted continuation callables or become the live registry merely because append-only storage is convenient. ## Public API direction Follow existing structured-outcome naming/style. Exact predicate names are intentionally left to implementation, but the public surface should provide semantics equivalent to: ```prolog scoped_state_store_open(+BackendSpec, -Outcome). scoped_state_store_close(+Store, -Outcome). scope_resolve(+ScopeSpec, +Options, -Outcome). project_identity(+ProjectHint, +Options, -Outcome). scoped_state_get(+Store, +Scope, +Key, -Outcome). scoped_state_list(+Store, +Scope, +Options, -Outcome). scoped_state_query(+Store, +Query, -Outcome). scoped_state_set(+Store, +TrustedMutation, -Outcome). scoped_state_revoke(+Store, +StateRefOrRuleId, +Reason, -Outcome). scoped_state_effective(+Store, +Scopes, +Key, +Options, -Outcome). scoped_state_status(+Store, +Ref, -Outcome). scoped_state_reload(+Store, -Outcome). ``` Names/arity may differ. The important contract is explicit store/scope identity, trusted mutation, inspectability, revocation, effective resolution, and backend independence. ### Trusted mutation boundary Mutation predicates are trusted host/library APIs, analogous to authority setters. They must not be registered as unrestricted model tools. A model can produce a **request** or candidate setting/policy as ordinary data. Converting that request into durable trusted state requires a mediated host/operator action. The API should make origin/trust explicit enough that loading arbitrary project/model data cannot masquerade as a host/operator mutation. ## Backend contract Provide at least: - deterministic in-memory backend for tests; - a persistent backend that survives process close/reopen/restart. The public semantics must not depend on SWI `library(persistency)` even if the first persistent implementation reuses that mechanism, as `rlm_artifact` and `rlm_graph` currently do. Backend requirements: - ground canonical data only; - deterministic open/close/reload behavior; - structured errors for malformed/corrupt/unsupported records; - schema/version checks before state becomes effective; - no implicit execution during loading; - atomic enough mutation semantics that a partially written revoke/update cannot accidentally leave a broader security policy active. If a future remote/latency-bearing backend is added, follow #54: canonical execute semantics -> async Future -> synchronous facade awaiting the same Future. Do **not** add fake async variants to pure/in-memory operations merely for visual symmetry. ## Precedence semantics This issue owns deterministic precedence for **ordinary non-security settings**. Define and test a documented scope overlay order. A likely shape is narrower scope overriding broader scope where eligible, but implementation must decide exact host/user/project/session ordering and how “unset”, revoked, invalid, and explicit values behave. Do **not** reuse ordinary overlay logic as the security-authority algorithm. Authority-affecting persisted rules are resolved by #76 through the #53 authority boundary and remain capped by current hard/host policy. This substrate stores and queries them but does not decide that they authorize execution. ## Security invariants - Model output cannot call an unrestricted durable mutation primitive. - Persisted records are closed validated data, not arbitrary executable Prolog. - No arbitrary loaded term is passed to unrestricted `call/1`. - Project files do not become trusted state merely because they were discovered. - Loading/reloading state cannot grant tool capabilities. - Loading/reloading state cannot set `dangerous` or make `allow_session` persistent. - Malformed, unknown-schema, ambiguous-project, or untrusted authority-affecting state fails closed. - Persistent values cannot bypass schemas, path confinement, process/network policy, budgets, cancellation, capabilities, or effect accounting. - Secrets are represented by references to trusted external configuration where needed; ordinary scoped records must not become a plaintext secret store by default. Preserve the separation hardened by #52/#72. - Audit/history does not retain executable continuations. ## Interaction with existing modules/issues ### #16 / `rlm_artifact` Reuse persistence/versioning lessons where useful, not artifact semantics. Artifacts are model/task knowledge and can be published by reasoning workflows; that must never imply permission mutation. ### `rlm_graph` / agent state Graph checkpoints and agent runtime state remain execution state. They may carry references to scoped state but are not the durable settings database. ### #53 / `rlm_authority` This issue does not persist authority modes and does not implement authorization decisions. #76 composes scoped policy with #53. ### #52 / `rlm_mcp_policy` MCP profile/config-reference facts are hard trusted lifecycle configuration. They remain separate from user/project preferences and remembered permissions. ### #68-#71 #70 may resolve effective operator/project policy as a context source, but its `rlm_artifact` snapshots remain epistemic resolved-context records. Preserve provenance classes rather than flattening the stores. ### #48/#49/#50 The core abstraction must be generic enough for external tool libraries and downstream agents without importing their concrete catalogs or UX. ## Acceptance criteria - [ ] Closed scope terms exist for host/user/project/session or documented repository-equivalent forms. - [ ] Project identity is structured and not equal to raw `cwd`. - [ ] Deterministic tests cover symlink/canonical-root behavior and at least explicit worktree/multiple-clone/non-Git identity decisions. - [ ] A memory backend and persistent backend expose the same record/read/list/revoke/effective semantics. - [ ] A persistent project setting survives close/reopen in a fresh Prolog process. - [ ] Project A and project B can hold the same key with different values without leakage. - [ ] Ordinary setting precedence across configured scopes is deterministic and documented. - [ ] Trusted mutation records provenance/origin and rejects non-ground/malformed values. - [ ] Revocation makes a record ineligible immediately and after restart. - [ ] Superseded/revoked records remain inspectable without remaining active. - [ ] Unknown schema/version or malformed persistent records fail closed and never become effective authority policy. - [ ] A model-produced term cannot directly mutate trusted durable state through any model-facing plan/tool surface. - [ ] Loading an arbitrary project file does not mutate the scoped store. - [ ] State reload does not change the current authority tier and never converts `allow_session` to persistent state. - [ ] No generic `fact/7` API collapses artifacts, repository context, MCP declarations, and trusted operator state into one semantic class. - [ ] Public exports and documentation clearly identify trusted mutation APIs versus read/inspection APIs. - [ ] Tests prove active state and audit/history are separable and no executable callback/continuation is retained for history. ## Non-goals - No persistent project authorization matching/decision logic in this issue; see #76. - No concrete project-local Prolog pack/file convention; see #77. - No coding-agent-specific preference schema. - No UI/TUI. - No arbitrary Prolog code execution from persisted values. - No general secret-management service. - No fake async API for pure operations. - Do not replace `rlm_artifact`, `rlm_context`, or `rlm_graph` storage with this substrate. ## Dependencies / references - Parent #74 - #16 durable artifacts - #48 external tool-library boundary - #52 completed MCP declarative/config policy - #53 host-controlled authority - #54 sync/async contract - #64 active-vs-terminal authority retention lesson - #68-#71 context/workflow pipeline - `research/RLM-RESEARCH-003-typed-symbolic-execution.org` - `research/RLM-RESEARCH-005-swi-agent-runtime.org` - `research/RLM-RESEARCH-007-langchain-langgraph-port.org` - `research/RLM-RESEARCH-009-durable-artifact-context.org` Blocks #76 and #77.