nsaspy/starintel-auto-research

Fork 0

DESIGN_READY_FOR_OPERATOR_REVIEW: StarLang Source Acquisition / Scraping Domain Server #169

New issue

Open

opened 2026-08-29 10:28:06 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-08-29 10:28:06 +00:00

Owner

Authority / state

Source research: #169
ARDR policy: #170
Research transition: READY_FOR_DESIGN
Design state: DESIGN_READY_FOR_OPERATOR_REVIEW
Implementation approval: PENDING — operator only

This design does not authorize implementation and must not enter the executable RAGE queue until the operator explicitly approves an implementation slice.

StarIntel problem solved

StarIntel currently has domain-server/remoting/runtime primitives but no coherent StarLang-owned source-acquisition definition surface. Without one, OSINT actors, expert systems, Auto-Research workers, search providers, and future data-source plugins risk duplicating HTTP/browser/feed/scraping stacks and diverging on provenance, cancellation, resource limits, and local-vs-remote semantics.

Owning boundaries

lost-rob0t/star-lang: scraper/source definition language, typed IR, lifecycle semantics, capability references, provenance propagation, bounded iteration primitives.
lost-rob0t/starintel-server: Source Acquisition Domain Server runtime integration and Common Lisp HTTP/DOM/feed adapters.
browser adapter: isolated capability adapter beneath StarLang; Chromium/Playwright-style process/protocol boundary is acceptable where browser execution is required.
lost-rob0t/starintel-auto-research: researcher integration and canonical research/design authority.
expert systems: select/ask for collection through ordinary actor/domain capability calls; expert rule RHS stays effect-closed.
Search/YaCy/SearX: downstream source definitions/providers, not architectural owners.

Existing runtime reused

Current StarLang already provides:

compiled domain servers with typed accepted messages, declared tools, bounded mailboxes and capabilities;
lifecycle command/reply/error/ack envelopes;
correlation and idempotency identity;
deadlines and cancellation;
terminal replay without re-executing completed work;
deferred completion;
remote domain-node registration and round-trip request/reply through the same lifecycle contract;
runtime journal events including pending, route-result, and remote-result.

Therefore local and remote acquisition use one operation contract. No second remote scraper API is designed.

Domain contract

Canonical domain: SourceAcquisitionDomainServer (final spelling may follow StarLang naming conventions).

Conceptual request:

acquire {
  scraperRef,
  input,
  principalRef,
  deadline,
  idempotencyKey?,
  sessionRef?,
  budget,
  artifactPolicy
}

Conceptual outputs:

accepted
item(observation/document/artifactRef, provenance)
warning(typedReason)
terminal(success|partial|blocked|unavailable|cancelled|failed, summary)

Large bodies, screenshots, browser state, credentials and retained captures travel by opaque scoped refs, never raw secrets/session material in scraper source or normal lifecycle envelopes.

StarLang scraper definition surface

Add a small declarative source-definition layer, not a second programming language.

A scraper definition owns:

stable name/version identity;
accepted input type;
emitted typed schema/document types;
request/navigation steps;
extraction declarations;
bounded iteration/pagination policy;
optional capability requirements;
normalization bindings using ordinary StarLang expressions where possible;
fixture identity/hash;
provenance inheritance policy.

Conceptual shape only:

(scraper searx-search
  (:version "1"
   :input search-query
   :emits search-result
   :requires (http))

  (request GET endpoint
    (:query ((q query)
             (format "json")
             (pageno page))))

  (paginate
    (:kind page :start 1 :max-pages 10)
    ...)

  (extract-json
    (:each "results")
    (emit search-result
      ...)))

The final grammar must lower into closed typed IR; scraper source may not name arbitrary host functions, shell commands, native Playwright objects, or raw credentials.

Required StarLang extensions

Slice A — scraper definition + typed acquisition IR

Needed in StarLang:

scraper/source declaration kind;
stable scraper version/fixture identity;
typed request/response/item/terminal IR;
capability references (http, dom, browser, feed) without embedding implementation objects;
output schema binding;
provenance edges from request -> response/artifact -> extraction -> emitted item.

Slice B — bounded iteration / traversal

Needed in StarLang/runtime:

page/offset/cursor/Link-token pagination forms;
explicit maximum iterations/items/bytes/time;
visited/request-fingerprint set;
cycle detection;
partial terminal outcome on policy/resource exhaustion;
no implicit infinite crawl.

Slice C — opaque references + optional capability continuation

Needed in StarLang/runtime if not already generalized elsewhere:

principal-scoped opaque credentialRef, sessionRef, artifactRef, resultRef values;
expiry/fencing semantics;
optional capability discovery;
typed blockedByChallenge / unsupportedChallenge / partial continuation compatible with #168;
resumed acquisition preserves original causation/provenance.

Slice D — incremental item emission

Current command handlers mainly model terminal reply/ack. Acquisition needs bounded intermediate item observations while preserving one final terminal outcome. Add a typed lifecycle projection for incremental emitted items or define the existing actor-message mechanism as the canonical stream path. Do not create an unbounded side channel.

Lower adapter contracts

HTTP adapter

Common Lisp implementation beneath StarLang should expose typed operations for:

method/URL/query/headers/body;
content length/decoded size ceilings;
redirects with policy revalidation on every hop;
TLS policy;
DNS resolution with private/metadata/link-local restrictions;
content type/status classification;
deadlines/cancellation;
bounded retry/backoff/Retry-After;
per-host/global concurrency and rate policy;
cache/revalidation policy;
proxy/transport selection;
opaque credential/session refs.

DOM adapter

Typed parse/query boundary supporting:

HTML/XML parse with parser/resource ceilings;
CSS selectors;
XPath where the selected CL library supports it cleanly, otherwise a documented equivalent plus an XPath adapter capability;
text/attribute extraction;
nested/repeating extraction;
explicit missing/ambiguous/parse failure diagnostics.

Feed adapter

Feed support can remain a library/adapter pattern rather than new syntax if typed iteration is sufficient.

Normalize:

RSS channel/feed metadata;
RSS item identity: prefer guid when valid/stable; otherwise use deterministic source-specific fallback from link + publication/content hash with provenance showing the fallback;
Atom feed/entry atom:id as stable identity;
published and updated separately;
links/alternate/self/enclosures;
extension namespaces without silently flattening unknown fields into authority;
incremental polling with conditional HTTP/caching and deterministic dedup/change detection.

Browser adapter

Use an isolated browser capability beneath StarLang. A Chromium/Playwright process/service adapter is justified because current Playwright provides isolated BrowserContexts, navigation/action auto-waiting, page/frame interaction, cookie/session handling and context-level network routing/interception.

Required StarIntel boundary:

one principal-scoped browser context/session ref per acquisition isolation domain;
page count, memory/CPU/time and navigation ceilings;
navigate/wait/click/fill/select/evaluate only through closed typed operations;
JS evaluation disabled by default unless scraper capability explicitly requires it;
screenshot/HAR/body capture only on explicit artifact policy;
browser network requests subject to the same target policy as HTTP acquisition;
service-worker/network interception limitations recorded in typed diagnostics;
close/fence context on cancellation/expiry.

Downstream proof definitions

Static HTML

HTTP GET -> bounded body -> DOM parse -> CSS/XPath extraction -> normalized typed items -> provenance -> terminal summary.

JavaScript-rendered page

Acquire browser capability -> create scoped context -> navigate -> wait for declared condition -> interaction steps -> extract rendered DOM -> emit typed items -> close/fence context -> terminal summary.

Paginated JSON API

HTTP request template -> JSON parse -> emit results -> obtain page/cursor token -> bounded paginate -> partial on exhausted budget or malformed cursor.

RSS / Atom

Conditional GET -> parse feed -> normalize stable item/entry identity + timestamps + links/enclosures -> dedup against prior observation identity -> emit new/changed entries -> terminal summary.

SearX/SearXNG

Use /search or / with configured machine-readable format, query/category/language/page parameters. Treat JSON/RSS support as instance capability because formats may be disabled. A 403/unsupported format becomes typed unsupported/unavailable, not an HTML parsing guess. Preserve instance and engine/provider provenance when present.

YaCy

Use the instance HTTP/JSON search boundary such as yacysearch.json with query parameters. Treat endpoint/version/capability as instance configuration and preserve YaCy instance + underlying result URL provenance. Pagination/record offsets are bounded and capability-tested; do not assume arbitrarily deep offsets are supported.

Expert-system integration

Expert rules remain effect-free. The expert may conclude evidence-needed(...) or select a source strategy; the owning actor/researcher turns that conclusion into an acquisition command. Returned typed observations are projected back into expert facts with stable source object/provenance identity. Partial/blocked acquisition is evidence state, not silently equivalent to absence/not-found.

Agentic researcher integration

Current Auto-Research deep-research architecture already treats external research engines/browser workers as bounded adapters under the Research Run Supervisor and Branch Supervisor; adapters receive scoped inputs/budgets and return normalized source/evidence events. Source Acquisition becomes the reusable collection capability underneath those workers:

Research Run Supervisor
  -> researcher/adapter
  -> Source Acquisition capability
  -> scraper definition
  -> adapter primitive
  -> normalized evidence/item events
  -> researcher result

The acquisition domain never owns candidate promotion, research-plan authority, or canonical Org writes.

Security/resource invariants

SSRF policy validates resolved target before connect and every redirect; private, metadata, link-local and denied ranges fail closed unless explicit trusted policy grants them.
DNS rebinding: connect to a validated resolved address set and revalidate on re-resolution/redirect rather than trusting hostname text alone.
decompress/content/parser ceilings apply before unbounded allocation.
browser contexts and cookies never cross principal/session ownership.
no raw credential/session secrets in scraper definitions, logs, provenance, or fixtures.
effectful browser interactions are not automatically replay-safe; retry requires an explicitly safe step or a new fenced session/strategy.
pagination, recursive following, fan-out, retries and browser pages are all bounded.
unknown feed/XML extensions remain untrusted data.
cancellation closes/fences network/browser resources and prevents late completion from mutating newer run state.
raw source/artifact provenance remains separable from normalized extracted items.

Adversarial review

Rejected alternatives

Embed Scrapy as the architecture. Rejected: imports Twisted/Python lifecycle and duplicates StarLang actor/runtime ownership.
One actor per source/provider. Rejected: duplicates acquisition/resource/security semantics.
Let expert rules call HTTP/browser directly. Rejected: violates effect-closed expert semantics and creates confused-deputy/SSRF risk.
Browser worker owns research state. Rejected: conflicts with existing Auto-Research supervisor/ledger boundary.
Assume local and remote scraper APIs differ. Rejected: current StarLang remoting proves the same lifecycle contract can round-trip remotely with idempotent terminal replay.
Treat retry as generic replay. Rejected for clicks/forms/side-effecting interactions.
Assume SearXNG JSON or YaCy deep offsets always exist. Rejected: both are deployment/capability concerns.

Remaining design risks carried as implementation requirements

incremental item streaming must be defined without creating an unbounded channel;
exact DOM/XPath Common Lisp library selection remains an adapter implementation choice;
browser adapter protocol/library selection must prove cancellation/resource cleanup and principal isolation;
persisted crawl/visited state must reuse current StarLang journal/state ownership rather than invent a second durable scheduler.

No unresolved issue above requires changing the architecture boundary.

Dependency-ordered proposed implementation slices

All remain AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL.

StarLang scraper declaration + closed typed acquisition IR + provenance/fixture identity.
StarLang bounded iteration/visited-set + incremental item lifecycle semantics.
Opaque scoped refs + optional capability/challenge continuation (coordinate with #168).
Common Lisp HTTP + DOM + feed adapters behind the typed capability ports.
Browser adapter with isolated Chromium/Playwright-style process boundary and bounded lifecycle.
starintel-server Source Acquisition Domain Server integration using the same local/remote lifecycle contract.
SearXNG and YaCy source definitions/providers.
Auto-Research/expert-system integration fixtures.

RED-first targets

Every coding slice requires a failing contract test before production mutation. The first slice's RED target is a StarLang fixture containing a valid scraper declaration that must currently fail because the parser/compiler has no scraper declaration/typed acquisition IR. Invalid fixtures must prove arbitrary host calls/raw credentials/unbounded pagination are rejected by the closed grammar.

Implementation approval

PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL

ARDR has no authority to change this state.

## Authority / state - Source research: #169 - ARDR policy: #170 - Research transition: `READY_FOR_DESIGN` - Design state: `DESIGN_READY_FOR_OPERATOR_REVIEW` - **Implementation approval: PENDING — operator only** This design does not authorize implementation and must not enter the executable RAGE queue until the operator explicitly approves an implementation slice. ## StarIntel problem solved StarIntel currently has domain-server/remoting/runtime primitives but no coherent StarLang-owned source-acquisition definition surface. Without one, OSINT actors, expert systems, Auto-Research workers, search providers, and future data-source plugins risk duplicating HTTP/browser/feed/scraping stacks and diverging on provenance, cancellation, resource limits, and local-vs-remote semantics. ## Owning boundaries - `lost-rob0t/star-lang`: scraper/source definition language, typed IR, lifecycle semantics, capability references, provenance propagation, bounded iteration primitives. - `lost-rob0t/starintel-server`: Source Acquisition Domain Server runtime integration and Common Lisp HTTP/DOM/feed adapters. - browser adapter: isolated capability adapter beneath StarLang; Chromium/Playwright-style process/protocol boundary is acceptable where browser execution is required. - `lost-rob0t/starintel-auto-research`: researcher integration and canonical research/design authority. - expert systems: select/ask for collection through ordinary actor/domain capability calls; expert rule RHS stays effect-closed. - Search/YaCy/SearX: downstream source definitions/providers, not architectural owners. ## Existing runtime reused Current StarLang already provides: - compiled domain servers with typed accepted messages, declared tools, bounded mailboxes and capabilities; - lifecycle command/reply/error/ack envelopes; - correlation and idempotency identity; - deadlines and cancellation; - terminal replay without re-executing completed work; - deferred completion; - remote domain-node registration and round-trip request/reply through the same lifecycle contract; - runtime journal events including `pending`, `route-result`, and `remote-result`. Therefore local and remote acquisition use one operation contract. No second remote scraper API is designed. ## Domain contract Canonical domain: `SourceAcquisitionDomainServer` (final spelling may follow StarLang naming conventions). Conceptual request: ```text acquire { scraperRef, input, principalRef, deadline, idempotencyKey?, sessionRef?, budget, artifactPolicy } ``` Conceptual outputs: ```text accepted item(observation/document/artifactRef, provenance) warning(typedReason) terminal(success|partial|blocked|unavailable|cancelled|failed, summary) ``` Large bodies, screenshots, browser state, credentials and retained captures travel by opaque scoped refs, never raw secrets/session material in scraper source or normal lifecycle envelopes. ## StarLang scraper definition surface Add a small declarative source-definition layer, not a second programming language. A scraper definition owns: - stable name/version identity; - accepted input type; - emitted typed schema/document types; - request/navigation steps; - extraction declarations; - bounded iteration/pagination policy; - optional capability requirements; - normalization bindings using ordinary StarLang expressions where possible; - fixture identity/hash; - provenance inheritance policy. Conceptual shape only: ```lisp (scraper searx-search (:version "1" :input search-query :emits search-result :requires (http)) (request GET endpoint (:query ((q query) (format "json") (pageno page)))) (paginate (:kind page :start 1 :max-pages 10) ...) (extract-json (:each "results") (emit search-result ...))) ``` The final grammar must lower into closed typed IR; scraper source may not name arbitrary host functions, shell commands, native Playwright objects, or raw credentials. ## Required StarLang extensions ### Slice A — scraper definition + typed acquisition IR Needed in StarLang: - `scraper`/source declaration kind; - stable scraper version/fixture identity; - typed request/response/item/terminal IR; - capability references (`http`, `dom`, `browser`, `feed`) without embedding implementation objects; - output schema binding; - provenance edges from request -> response/artifact -> extraction -> emitted item. ### Slice B — bounded iteration / traversal Needed in StarLang/runtime: - page/offset/cursor/Link-token pagination forms; - explicit maximum iterations/items/bytes/time; - visited/request-fingerprint set; - cycle detection; - partial terminal outcome on policy/resource exhaustion; - no implicit infinite crawl. ### Slice C — opaque references + optional capability continuation Needed in StarLang/runtime if not already generalized elsewhere: - principal-scoped opaque `credentialRef`, `sessionRef`, `artifactRef`, `resultRef` values; - expiry/fencing semantics; - optional capability discovery; - typed `blockedByChallenge` / `unsupportedChallenge` / `partial` continuation compatible with #168; - resumed acquisition preserves original causation/provenance. ### Slice D — incremental item emission Current command handlers mainly model terminal reply/ack. Acquisition needs bounded intermediate item observations while preserving one final terminal outcome. Add a typed lifecycle projection for incremental emitted items or define the existing actor-message mechanism as the canonical stream path. Do not create an unbounded side channel. ## Lower adapter contracts ### HTTP adapter Common Lisp implementation beneath StarLang should expose typed operations for: - method/URL/query/headers/body; - content length/decoded size ceilings; - redirects with policy revalidation on every hop; - TLS policy; - DNS resolution with private/metadata/link-local restrictions; - content type/status classification; - deadlines/cancellation; - bounded retry/backoff/Retry-After; - per-host/global concurrency and rate policy; - cache/revalidation policy; - proxy/transport selection; - opaque credential/session refs. ### DOM adapter Typed parse/query boundary supporting: - HTML/XML parse with parser/resource ceilings; - CSS selectors; - XPath where the selected CL library supports it cleanly, otherwise a documented equivalent plus an XPath adapter capability; - text/attribute extraction; - nested/repeating extraction; - explicit missing/ambiguous/parse failure diagnostics. ### Feed adapter Feed support can remain a library/adapter pattern rather than new syntax if typed iteration is sufficient. Normalize: - RSS channel/feed metadata; - RSS item identity: prefer `guid` when valid/stable; otherwise use deterministic source-specific fallback from link + publication/content hash with provenance showing the fallback; - Atom feed/entry `atom:id` as stable identity; - `published` and `updated` separately; - links/alternate/self/enclosures; - extension namespaces without silently flattening unknown fields into authority; - incremental polling with conditional HTTP/caching and deterministic dedup/change detection. ### Browser adapter Use an isolated browser capability beneath StarLang. A Chromium/Playwright process/service adapter is justified because current Playwright provides isolated BrowserContexts, navigation/action auto-waiting, page/frame interaction, cookie/session handling and context-level network routing/interception. Required StarIntel boundary: - one principal-scoped browser context/session ref per acquisition isolation domain; - page count, memory/CPU/time and navigation ceilings; - navigate/wait/click/fill/select/evaluate only through closed typed operations; - JS evaluation disabled by default unless scraper capability explicitly requires it; - screenshot/HAR/body capture only on explicit artifact policy; - browser network requests subject to the same target policy as HTTP acquisition; - service-worker/network interception limitations recorded in typed diagnostics; - close/fence context on cancellation/expiry. ## Downstream proof definitions ### Static HTML HTTP GET -> bounded body -> DOM parse -> CSS/XPath extraction -> normalized typed items -> provenance -> terminal summary. ### JavaScript-rendered page Acquire browser capability -> create scoped context -> navigate -> wait for declared condition -> interaction steps -> extract rendered DOM -> emit typed items -> close/fence context -> terminal summary. ### Paginated JSON API HTTP request template -> JSON parse -> emit results -> obtain page/cursor token -> bounded paginate -> partial on exhausted budget or malformed cursor. ### RSS / Atom Conditional GET -> parse feed -> normalize stable item/entry identity + timestamps + links/enclosures -> dedup against prior observation identity -> emit new/changed entries -> terminal summary. ### SearX/SearXNG Use `/search` or `/` with configured machine-readable format, query/category/language/page parameters. Treat JSON/RSS support as instance capability because formats may be disabled. A 403/unsupported format becomes typed `unsupported`/`unavailable`, not an HTML parsing guess. Preserve instance and engine/provider provenance when present. ### YaCy Use the instance HTTP/JSON search boundary such as `yacysearch.json` with query parameters. Treat endpoint/version/capability as instance configuration and preserve YaCy instance + underlying result URL provenance. Pagination/record offsets are bounded and capability-tested; do not assume arbitrarily deep offsets are supported. ## Expert-system integration Expert rules remain effect-free. The expert may conclude `evidence-needed(...)` or select a source strategy; the owning actor/researcher turns that conclusion into an acquisition command. Returned typed observations are projected back into expert facts with stable source object/provenance identity. Partial/blocked acquisition is evidence state, not silently equivalent to absence/not-found. ## Agentic researcher integration Current Auto-Research deep-research architecture already treats external research engines/browser workers as bounded adapters under the Research Run Supervisor and Branch Supervisor; adapters receive scoped inputs/budgets and return normalized source/evidence events. Source Acquisition becomes the reusable collection capability underneath those workers: ```text Research Run Supervisor -> researcher/adapter -> Source Acquisition capability -> scraper definition -> adapter primitive -> normalized evidence/item events -> researcher result ``` The acquisition domain never owns candidate promotion, research-plan authority, or canonical Org writes. ## Security/resource invariants - SSRF policy validates resolved target before connect and every redirect; private, metadata, link-local and denied ranges fail closed unless explicit trusted policy grants them. - DNS rebinding: connect to a validated resolved address set and revalidate on re-resolution/redirect rather than trusting hostname text alone. - decompress/content/parser ceilings apply before unbounded allocation. - browser contexts and cookies never cross principal/session ownership. - no raw credential/session secrets in scraper definitions, logs, provenance, or fixtures. - effectful browser interactions are not automatically replay-safe; retry requires an explicitly safe step or a new fenced session/strategy. - pagination, recursive following, fan-out, retries and browser pages are all bounded. - unknown feed/XML extensions remain untrusted data. - cancellation closes/fences network/browser resources and prevents late completion from mutating newer run state. - raw source/artifact provenance remains separable from normalized extracted items. ## Adversarial review ### Rejected alternatives 1. **Embed Scrapy as the architecture.** Rejected: imports Twisted/Python lifecycle and duplicates StarLang actor/runtime ownership. 2. **One actor per source/provider.** Rejected: duplicates acquisition/resource/security semantics. 3. **Let expert rules call HTTP/browser directly.** Rejected: violates effect-closed expert semantics and creates confused-deputy/SSRF risk. 4. **Browser worker owns research state.** Rejected: conflicts with existing Auto-Research supervisor/ledger boundary. 5. **Assume local and remote scraper APIs differ.** Rejected: current StarLang remoting proves the same lifecycle contract can round-trip remotely with idempotent terminal replay. 6. **Treat retry as generic replay.** Rejected for clicks/forms/side-effecting interactions. 7. **Assume SearXNG JSON or YaCy deep offsets always exist.** Rejected: both are deployment/capability concerns. ### Remaining design risks carried as implementation requirements - incremental item streaming must be defined without creating an unbounded channel; - exact DOM/XPath Common Lisp library selection remains an adapter implementation choice; - browser adapter protocol/library selection must prove cancellation/resource cleanup and principal isolation; - persisted crawl/visited state must reuse current StarLang journal/state ownership rather than invent a second durable scheduler. No unresolved issue above requires changing the architecture boundary. ## Dependency-ordered proposed implementation slices All remain **AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL**. 1. StarLang scraper declaration + closed typed acquisition IR + provenance/fixture identity. 2. StarLang bounded iteration/visited-set + incremental item lifecycle semantics. 3. Opaque scoped refs + optional capability/challenge continuation (coordinate with #168). 4. Common Lisp HTTP + DOM + feed adapters behind the typed capability ports. 5. Browser adapter with isolated Chromium/Playwright-style process boundary and bounded lifecycle. 6. `starintel-server` Source Acquisition Domain Server integration using the same local/remote lifecycle contract. 7. SearXNG and YaCy source definitions/providers. 8. Auto-Research/expert-system integration fixtures. ## RED-first targets Every coding slice requires a failing contract test before production mutation. The first slice's RED target is a StarLang fixture containing a valid `scraper` declaration that must currently fail because the parser/compiler has no scraper declaration/typed acquisition IR. Invalid fixtures must prove arbitrary host calls/raw credentials/unbounded pagination are rejected by the closed grammar. ## Implementation approval `PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL` ARDR has no authority to change this state.

nsaspy referenced this issue

2026-08-29 10:28:18 +00:00

Research + design: StarLang-defined scraping domain actor and researcher integration #171

nsaspy referenced this issue

2026-08-29 10:28:18 +00:00

Research + design: StarLang-defined scraping domain actor and researcher integration #171