nsaspy/starintel-server

Fork 0

[P1-06] Replace dead matcher/URL extractor with bounded typed document transforms #134

New issue

Open

opened 2026-09-02 17:26:32 +00:00 by nsaspy · 1 comment

nsaspy commented

2026-09-02 17:26:32 +00:00

Owner

ARDR reconciliation — 2026-08-29

Canonical research/design now own this issue:

Research: lost-rob0t/starintel-auto-research/roam/research/star-server/STAR-RESEARCH-051-document-pattern-ingress-url-extraction.org — READY_FOR_DESIGN.
Design/adversarial review: lost-rob0t/starintel-auto-research/roam/design/star-server/STAR-SERVER-050-document-pattern-ingress-url-extraction.org — DESIGN_READY_FOR_OPERATOR_REVIEW.
Auto-Research transaction: lost-rob0t/starintel-auto-research#200.
Implementation authorization: PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL.

Current-source findings

Current master loads source/actor-systems/matcher-actor.lisp, and start-url-extractor registers a pattern in global star:*document-patterns*, but current source has no document-ingress component that consumes canonical incoming documents and iterates that registry. The registered matcher is therefore operationally dead.

The legacy path also:

stores arbitrary Common Lisp match-fn callbacks in mutable global process state;
stores raw actor refs/names as subscriptions;
scans only hard-coded content/bio fields;
uses an ad-hoc URL regex as both candidate detector and URI validator;
publishes generated URL/relation documents directly to documents.new.* through the global producer agent;
bypasses canonical application ingest, validation/persistence, durable outbox semantics, and durable idempotency;
has no bounded candidate/output/deadline contract or per-transform failure isolation.

Current server persistence already has mutation IDs and durable outbox/event mechanics, and the server dependency closure already contains quri. Do not add another process merely to parse URI candidates.

Design decision

Replace the legacy global matcher registry as authority with one StarLang-first typed document-transform boundary linked through the #43 RuntimePlan.

validated canonical document
        ↓
RuntimePlan transform binding
        ↓
bounded transform executor
        ↓
typed generated documents + provenance
        ↓
canonical application/ingest port
        ↓
validation + idempotency + persistence/outbox
        ↓
documents.new.* event publication

The transform executor never publishes authoritative document events directly.

Transform definitions

Transform definitions are immutable/versioned and may reference closed StarLang semantics or trusted installed adapter IDs. They may not contain arbitrary Lisp callbacks, shell commands, raw actor refs, Rabbit routing keys, credentials, or live database/broker objects.

Dataset/actor manifests may select or narrow compatible transforms, but cannot widen StarLang-declared input/output types or capabilities.

Runtime identity and bounds

#42 component handles/generations own runtime destination identity.
#34/#35/#36 remain thread/Sento/admission authorities.
Every execution has bounded input bytes, candidates, outputs, aggregate outputs, and one propagated absolute deadline.
One transform failure is isolated and represented as a typed partial/failure outcome.
Cancellation/stale execution cannot commit through a superseded component generation.

URL transform v1

The first vertical is extract.http-url.v1:

inspect only explicitly configured typed text selectors;
bounded lexical discovery recognizes explicit http:// / https:// starts;
conservatively trim surrounding prose punctuation;
parse candidate URI through the in-process URI library;
require HTTP(S), authority, and non-empty host;
apply deterministic fixture-defined normalization without erasing meaningful path/query distinctions;
deduplicate normalized candidates within the source document;
construct canonical URL + source→URL relation documents with source field/span and transform provenance;
submit generated documents through canonical ingest;
do not dereference/fetch URLs here — network acquisition belongs to Source Acquisition.

Extraction proves only that the source text contained the URI candidate. It does not establish reachability, ownership, identity, or corroboration.

Dependencies

Research/design can proceed now; production realization is dependency-ordered:

#43 RuntimePlan/manifest linking;
#40/#41 explicit application ports and concrete adapters;
#42 component handles/generation fencing;
#36 bounded admission;
#47 durable end-to-end idempotency for retry/redelivery semantics;
existing durable persistence/outbox path remains canonical.

Do not recreate local substitutes for those pending seams.

Proposed implementation slices — ALL PENDING OPERATOR APPROVAL

Slice 1 — pure transform contract + URL fixture corpus

No actor/network/database/Rabbit effects.

Mandatory RED-first target:

source text:
  "see https://Example.COM/a?x=1, then continue"

extract.http-url.v1(maxCandidates=8,maxOutputs=8)
=> https://example.com/a?x=1
=> exact source field/span provenance
=> publish/persist/network callback counts = 0

Untouched server must fail because no typed pure transform contract/normalizer exists. Calling the legacy regex does not count as GREEN.

Sibling REDs: malformed percent escapes, punctuation, balanced parentheses, explicit ports, IPv6 literals, Unicode/IDNA policy, duplicate normalized candidates, oversized input, max-candidate/max-output truncation, and schemes outside HTTP(S).

Slice 2 — fake canonical-ingest executor

generated URL/relation docs flow only through a fake typed ingest port;
one failing transform does not suppress a successful sibling;
limit/deadline outcomes are explicit;
zero direct Rabbit calls.

Slice 3 — RuntimePlan binding

After #43 realization, reject invalid dtype/output/capability bindings before actor/thread/network realization.

Slice 4 — first real canonical-ingest vertical

After #40/#41 and the relevant #47 idempotency seam, persist URL/relation through the validated mutation/outbox path.

Slice 5 — lifecycle/component-handle integration

After #42/#37 realization, restart gives a new generation and a stale handle cannot invoke the replacement.

Slice 6 — legacy retirement

After equivalence/restart evidence, remove/delegate *document-patterns*, raw callback/subscription authority, and direct URL extractor publication. Never run legacy direct-publish and new canonical-ingest extraction simultaneously for the same flow.

Acceptance criteria

configured validated documents actually reach the transform pipeline;
transform semantics are immutable, typed, StarLang-first and manifest-linked;
no arbitrary host callback or raw actor ref in transform configuration;
URL extraction uses bounded lexical candidate discovery plus URI parsing/normalization;
deterministic corpus covers edge cases and normalization policy;
generated URL/relation documents enter canonical ingest with provenance;
migrated path performs no direct transform-to-documents.new.* publication;
per-transform and aggregate limits/deadline are enforced;
one transform failure is isolated;
local duplicate candidates collapse; durable retry dedupe remains canonical idempotency's job;
stale component handles fail closed;
extraction does not imply reachability/ownership/corroboration;
no product implementation begins until explicit operator approval is recorded.

Implementation authorization

PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL

P1 priority, READY_FOR_DESIGN, completed design/adversarial review, issue state, publication, or machine output do not authorize RAGE implementation.

## ARDR reconciliation — 2026-08-29 Canonical research/design now own this issue: - Research: `lost-rob0t/starintel-auto-research/roam/research/star-server/STAR-RESEARCH-051-document-pattern-ingress-url-extraction.org` — `READY_FOR_DESIGN`. - Design/adversarial review: `lost-rob0t/starintel-auto-research/roam/design/star-server/STAR-SERVER-050-document-pattern-ingress-url-extraction.org` — `DESIGN_READY_FOR_OPERATOR_REVIEW`. - Auto-Research transaction: `lost-rob0t/starintel-auto-research#200`. - **Implementation authorization: PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL.** ## Current-source findings Current `master` loads `source/actor-systems/matcher-actor.lisp`, and `start-url-extractor` registers a pattern in global `star:*document-patterns*`, but current source has no document-ingress component that consumes canonical incoming documents and iterates that registry. The registered matcher is therefore operationally dead. The legacy path also: - stores arbitrary Common Lisp `match-fn` callbacks in mutable global process state; - stores raw actor refs/names as subscriptions; - scans only hard-coded `content`/`bio` fields; - uses an ad-hoc URL regex as both candidate detector and URI validator; - publishes generated URL/relation documents directly to `documents.new.*` through the global producer agent; - bypasses canonical application ingest, validation/persistence, durable outbox semantics, and durable idempotency; - has no bounded candidate/output/deadline contract or per-transform failure isolation. Current server persistence already has mutation IDs and durable outbox/event mechanics, and the server dependency closure already contains `quri`. Do not add another process merely to parse URI candidates. ## Design decision Replace the legacy global matcher registry as authority with one **StarLang-first typed document-transform boundary** linked through the #43 RuntimePlan. ```text validated canonical document ↓ RuntimePlan transform binding ↓ bounded transform executor ↓ typed generated documents + provenance ↓ canonical application/ingest port ↓ validation + idempotency + persistence/outbox ↓ documents.new.* event publication ``` The transform executor never publishes authoritative document events directly. ### Transform definitions Transform definitions are immutable/versioned and may reference closed StarLang semantics or trusted installed adapter IDs. They may not contain arbitrary Lisp callbacks, shell commands, raw actor refs, Rabbit routing keys, credentials, or live database/broker objects. Dataset/actor manifests may select or narrow compatible transforms, but cannot widen StarLang-declared input/output types or capabilities. ### Runtime identity and bounds - #42 component handles/generations own runtime destination identity. - #34/#35/#36 remain thread/Sento/admission authorities. - Every execution has bounded input bytes, candidates, outputs, aggregate outputs, and one propagated absolute deadline. - One transform failure is isolated and represented as a typed partial/failure outcome. - Cancellation/stale execution cannot commit through a superseded component generation. ## URL transform v1 The first vertical is `extract.http-url.v1`: 1. inspect only explicitly configured typed text selectors; 2. bounded lexical discovery recognizes explicit `http://` / `https://` starts; 3. conservatively trim surrounding prose punctuation; 4. parse candidate URI through the in-process URI library; 5. require HTTP(S), authority, and non-empty host; 6. apply deterministic fixture-defined normalization without erasing meaningful path/query distinctions; 7. deduplicate normalized candidates within the source document; 8. construct canonical URL + source→URL relation documents with source field/span and transform provenance; 9. submit generated documents through canonical ingest; 10. do **not** dereference/fetch URLs here — network acquisition belongs to Source Acquisition. Extraction proves only that the source text contained the URI candidate. It does not establish reachability, ownership, identity, or corroboration. ## Dependencies Research/design can proceed now; production realization is dependency-ordered: - #43 RuntimePlan/manifest linking; - #40/#41 explicit application ports and concrete adapters; - #42 component handles/generation fencing; - #36 bounded admission; - #47 durable end-to-end idempotency for retry/redelivery semantics; - existing durable persistence/outbox path remains canonical. Do not recreate local substitutes for those pending seams. ## Proposed implementation slices — ALL PENDING OPERATOR APPROVAL ### Slice 1 — pure transform contract + URL fixture corpus No actor/network/database/Rabbit effects. **Mandatory RED-first target:** ```text source text: "see https://Example.COM/a?x=1, then continue" extract.http-url.v1(maxCandidates=8,maxOutputs=8) => https://example.com/a?x=1 => exact source field/span provenance => publish/persist/network callback counts = 0 ``` Untouched server must fail because no typed pure transform contract/normalizer exists. Calling the legacy regex does not count as GREEN. Sibling REDs: malformed percent escapes, punctuation, balanced parentheses, explicit ports, IPv6 literals, Unicode/IDNA policy, duplicate normalized candidates, oversized input, max-candidate/max-output truncation, and schemes outside HTTP(S). ### Slice 2 — fake canonical-ingest executor - generated URL/relation docs flow only through a fake typed ingest port; - one failing transform does not suppress a successful sibling; - limit/deadline outcomes are explicit; - zero direct Rabbit calls. ### Slice 3 — RuntimePlan binding After #43 realization, reject invalid dtype/output/capability bindings before actor/thread/network realization. ### Slice 4 — first real canonical-ingest vertical After #40/#41 and the relevant #47 idempotency seam, persist URL/relation through the validated mutation/outbox path. ### Slice 5 — lifecycle/component-handle integration After #42/#37 realization, restart gives a new generation and a stale handle cannot invoke the replacement. ### Slice 6 — legacy retirement After equivalence/restart evidence, remove/delegate `*document-patterns*`, raw callback/subscription authority, and direct URL extractor publication. Never run legacy direct-publish and new canonical-ingest extraction simultaneously for the same flow. ## Acceptance criteria - configured validated documents actually reach the transform pipeline; - transform semantics are immutable, typed, StarLang-first and manifest-linked; - no arbitrary host callback or raw actor ref in transform configuration; - URL extraction uses bounded lexical candidate discovery plus URI parsing/normalization; - deterministic corpus covers edge cases and normalization policy; - generated URL/relation documents enter canonical ingest with provenance; - migrated path performs no direct transform-to-`documents.new.*` publication; - per-transform and aggregate limits/deadline are enforced; - one transform failure is isolated; - local duplicate candidates collapse; durable retry dedupe remains canonical idempotency's job; - stale component handles fail closed; - extraction does not imply reachability/ownership/corroboration; - no product implementation begins until explicit operator approval is recorded. ## Implementation authorization `PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL` P1 priority, `READY_FOR_DESIGN`, completed design/adversarial review, issue state, publication, or machine output do **not** authorize RAGE implementation.

nsaspy commented

2026-09-02 17:26:52 +00:00

Author

Owner

Operator direction update for STAR-RESEARCH-051:

document matcher definitions and matcher evaluation semantics move into StarLang;
starintel-server must not replace *document-patterns* with another Lisp callback/regex registry or a second matcher DSL;
the server embeds final StarLang systems in-process, supplies typed bounded application/effect ports, and remains authoritative for canonical ingest/persistence/resource admission;
the first URL matcher/transform is a bounded StarLang program that emits typed URL + relation candidates with provenance, then routes them through canonical ingest;
direct Rabbit publication is still not proof of ingest acceptance.

StarLang embedding readiness is now tracked by https://github.com/lost-rob0t/star-lang/issues/53.

Current determination: StarLang is not yet ready as a stable production embedding boundary. Final starlang-runtime has real runtime/actor APIs, but the complete compiler/load/document public path is still prototype-owned and StarLang #6/#47/#52 remain open. Do not work around this by fossilizing prototype packages in the server; finish the final embedding boundary in StarLang first.

Canonical research update commit: github.com/lost-rob0t/starintel-auto-research@a90b8ee619

Operator direction update for `STAR-RESEARCH-051`: - document matcher definitions **and matcher evaluation semantics move into StarLang**; - `starintel-server` must not replace `*document-patterns*` with another Lisp callback/regex registry or a second matcher DSL; - the server embeds final StarLang systems in-process, supplies typed bounded application/effect ports, and remains authoritative for canonical ingest/persistence/resource admission; - the first URL matcher/transform is a bounded StarLang program that emits typed URL + relation candidates with provenance, then routes them through canonical ingest; - direct Rabbit publication is still not proof of ingest acceptance. StarLang embedding readiness is now tracked by https://github.com/lost-rob0t/star-lang/issues/53. Current determination: **StarLang is not yet ready as a stable production embedding boundary.** Final `starlang-runtime` has real runtime/actor APIs, but the complete compiler/load/document public path is still prototype-owned and StarLang #6/#47/#52 remain open. Do not work around this by fossilizing prototype packages in the server; finish the final embedding boundary in StarLang first. Canonical research update commit: https://github.com/lost-rob0t/starintel-auto-research/commit/a90b8ee6192d062f9e1618d3d04ed9a2dcb45877

nsaspy referenced this issue from a commit

2026-09-08 01:48:34 +00:00

Merge pull request #134 from lost-rob0t/fix/v09-false-value-pipeline