nsaspy/starintel-auto-research

Fork 0

Research + design: StarLang-defined scraping domain actor and researcher integration #171

New issue

Open

opened 2026-08-29 10:28:06 +00:00 by nsaspy · 2 comments

nsaspy commented

2026-08-29 10:28:06 +00:00

Owner

Operator decision

This scope is pre-approved to advance from research directly into design once the required research is complete and internally consistent. Do not stop for another approval gate between research and design unless the work would materially expand beyond this issue or discovers a real security/architecture blocker.

Research everything named below. Do not reduce this to “port Scrapy.” The goal is to determine the StarLang/runtime semantics needed so StarIntel can define reusable scrapers and expose them through a first-class actor/domain-server contract.

Goal

Research and design a StarIntel scraping capability, implemented StarLang-first, that lets StarLang define scrapers for:

JavaScript-rendered pages
ordinary HTML pages
HTTP APIs
RSS / Atom feeds
YaCy instances as an example downstream integration
SearX / SearXNG instances as an example downstream integration

The scraping actor/domain server must support request/reply to either a local or remote caller through the same typed contract. Locality is deployment/runtime topology, not a different API.

The resulting design should be reusable by StarIntel expert systems, agents, Auto-Research, agentic researcher actors, OSINT domain servers, and ordinary StarIntel actors without each subsystem inventing its own HTTP/browser stack.

Architecture direction

Treat this as a coherent Scraping / Source Acquisition Domain Server or equivalent domain boundary after research validates the exact name/ownership.

Do not create one actor per website or one actor per upstream scraping framework.

Expected shape:

caller actor / expert / researcher
  -> typed scrape request
  -> local or remote Scraping Domain Server
  -> StarLang-defined scraper
  -> HTTP / browser / feed / API adapter
  -> typed extracted records + provenance
  -> reply / streamed observations / terminal outcome

Provider-specific/browser-specific implementation details stay behind capability adapters.

1. Deeply research Scrapy

Research current Scrapy architecture and enumerate the reusable semantics StarIntel actually needs, including at minimum:

Spider definition model
start requests / request generation
Request / Response model
callbacks and errbacks
selectors and extraction
CSS/XPath semantics and selector composition
item/data models
item loaders where useful
pipelines / post-processing
downloader middleware
spider middleware
scheduler / request queue
duplicate filtering / request fingerprints
depth and traversal controls
redirects
cookies / sessions
authentication patterns
headers / user agents
retries / backoff
throttling / per-host concurrency
robots behavior and policy hooks
proxy support
DNS/network behavior relevant to scraping
HTTP caching
stateful/resumable jobs
persistence / job directories if still relevant
feed exports
signals / lifecycle hooks
extensions / plugin model
stats / instrumentation
error taxonomy
cancellation / shutdown behavior
async/concurrency architecture
browser/JavaScript integrations, especially the current Playwright ecosystem around Scrapy
API/JSON/XML extraction patterns
pagination/cursor traversal
sitemap/feed discovery where architecturally useful

Do not cargo-cult Twisted, Python classes, or Scrapy internals into StarLang. Extract the semantic contract.

2. Research what StarLang must support to DEFINE scrapers

Compare the requirements against the current lost-rob0t/star-lang compiler/runtime. For every item classify exactly one:

already supported
expressible as a StarLang library/pattern now
StarLang syntax/IR/runtime extension needed
Common Lisp adapter/runtime primitive needed below StarLang
isolated external process required

Research at least these language/runtime semantics:

Fetch / protocol

typed HTTP requests and responses
method, URL, query, headers, body
bounded response size
redirects
content-type dispatch
JSON, XML, HTML, text and binary/artifact refs
HTTP status/error classification
request deadlines
cancellation
bounded retry/backoff
per-host/global concurrency limits
rate limits and throttling
cache policy
proxy / transport selection
TLS policy
authenticated requests through opaque credential/session refs, never raw secrets in scraper source

HTML extraction

DOM/document parse primitive
CSS selectors
XPath or a justified equivalent
attribute/text extraction
nested/repeating extraction
normalization/transforms
typed result schemas
required vs optional fields
extraction failure diagnostics

JavaScript pages

Research the browser-backed capability needed for:

navigate
execute page JavaScript when authorized
wait for DOM/network conditions
select/extract after rendering
click/fill/follow actions when a scraper requires interaction
frames if needed
pagination / infinite scroll patterns
network interception where useful
browser session/cookie state via opaque refs
bounded browser CPU/RAM/time
cancellation and stale-session fencing
screenshot/artifact refs only when requested

Determine whether the correct implementation is a Common Lisp browser adapter, a Chrome/Playwright process adapter, another browser protocol adapter, or a combination. StarLang remains the scraper definition/control language.

APIs

Support reusable definitions for:

REST-ish JSON/XML APIs
query/body parameter templates
headers/auth refs
pagination (page, offset, cursor, Link header, custom token)
rate-limit observation
retry-after
schema extraction / normalization
typed API errors

RSS / Atom

Research first-class feed support or a small library pattern for:

RSS variants
Atom
feed metadata
entries/items
stable entry identity
publication/update timestamps
enclosures/links
incremental polling
dedup/change detection
provenance to the original feed and item

Scraper composition

Research a StarLang scraper-definition model for:

named scraper definitions
accepted input type
output schema/document type
steps / navigation / extraction
reusable extraction fragments
request templates
pagination/iteration
conditional branches
typed partial results
item emission during a run
terminal summary/result
provenance inheritance across request -> response -> extraction -> emitted document
deterministic scraper version identity
test fixtures / owned snapshots

Do not invent a second general-purpose programming language inside StarLang. Prefer a small declarative surface plus ordinary StarLang expressions/actors where needed.

3. Local + remote actor request/reply contract

Research and design one caller contract that works identically when the scraping actor/domain server is local or remote.

Required semantics:

capability discovery
typed request and reply
correlation ID
caller identity / authorization
deadline propagation
cancellation propagation
idempotency identity where applicable
streamed/intermediate item observations where useful
final terminal outcome
typed success | partial | blocked | unavailable | cancelled | failed-style outcomes based on current StarLang conventions
bounded payloads
opaque artifact/session/result refs for large/sensitive state
preserved provenance / causal chain
retry semantics that do not accidentally duplicate effectful interactions
transparent routing through current StarLang domain-server/remoting semantics rather than separate local/remote implementations

Cross-check existing approved StarLang research and current runtime rather than re-designing remoting from scratch.

4. CAPTCHA / challenge integration

Integrate research with #168.

A browser/HTTP scraper that encounters a challenge should be able to surface a typed challenge observation and optionally invoke captcha.solve through the capability/domain-server registry when policy permits.

Do not hard-code CAPTCHA providers or private solver knowledge into scraper definitions.

Preserve continuation state so the scrape can resume after challenge handling.

5. YaCy + SearX/SearXNG downstream examples

Use YaCy and SearX/SearXNG as example consumers/definitions, not the architectural boundary.

Research representative StarLang scraper/adapter definitions for:

YaCy

query API/result extraction
pagination
endpoint capability/config
failure/rate/timeout behavior
provenance indicating the YaCy instance + underlying result URL/source

SearX / SearXNG

machine-readable search endpoint(s)
engine/category parameters where supported
pagination
result normalization
provider/engine provenance when exposed
timeout/partial-result behavior

These examples should prove the same scraper primitives can express a real API/metasearch integration without writing a bespoke runtime actor for each service.

Coordinate conceptually with the StarIntel Search/provider work, but keep the scraping domain generic.

6. Research the StarIntel expert-system architecture

Research the current StarIntel expert-system direction and implementation, including the approved StarLang expert-system research referenced by #156.

Cover:

current StarLang expert-system language/runtime surface
expert-engine actor architecture
selectable logic backends (including Prolog where applicable)
fact/rule lifecycle
capability selection/delegation
explanation/proof traces
expert invocation from actors/domain servers
remote expert execution if supported/planned
how experts should select a scraper definition or scraping strategy
how scraped observations become typed facts/documents without bypassing provenance/validation
how scraper failures/partial results enter expert reasoning
how an expert asks for additional collection and receives the resulting evidence

Do not revive a Prolog-only StarIntel actor architecture. StarLang owns the expert-engine boundary; Prolog may be a backend.

7. Research StarIntel agents + agentic researcher actors

Research the current StarIntel actor/agent model and Auto-Research agentic-research architecture, including #45 and current implementation/research state.

Enumerate:

ordinary StarIntel actors vs researcher/agent actors
research-run supervisors/control plane
worker/adaptor boundaries
tool/capability discovery
actor spawning/delegation if present
researcher planning and evidence acquisition
cancellation/budgets/deadlines
result/provenance handling
durable run state
local vs remote workers
browser-backed workers
how expert systems participate in planning/selection
how the new scraping domain server should be called by researchers

Target architecture:

researcher/agent owns research intent/plan
expert system may decide/select collection strategy
scraping domain server owns source acquisition mechanics
scraper definition owns source-specific navigation/extraction
resulting observations/documents return through typed contracts with provenance

Do not let every agent ship its own fetch/browser/scraping implementation.

8. Security / resource model

Research and carry into design:

SSRF and private/metadata address policy
redirect-to-private-address handling
DNS rebinding considerations
browser sandboxing
untrusted HTML/JS
response/body size ceilings
download/artifact ceilings
CPU/RAM/browser page limits
per-host/global concurrency
rate-limit policy
credentials/session isolation
cookie isolation
cross-principal session leakage prevention
source allow/deny policy
audit/provenance without logging secrets/content unnecessarily
malicious feed/API payloads
decompression bombs
parser bombs / pathological DOMs
infinite pagination / crawl loops
visited-set / duplicate suppression
cancellation cleanup

9. Required research outputs

Create/refresh research nodes covering at minimum:

Scrapy architecture and reusable semantics
StarLang scraper-definition language/runtime gaps
browser/JavaScript scraping adapter requirements
HTTP/API/HTML/feed acquisition semantics
local/remote scraping actor request/reply contract
YaCy/SearX example implementations
StarIntel expert-system integration
StarIntel agents / agentic researcher integration
adversarial security/resource analysis

Update roam/internal/OSINT-TOOLS-LEDGER.org with Scrapy and any directly relevant browser/scraping technologies discovered during the pass.

Cross-reference #165, #168, #156, and #45 where applicable.

10. Direct-to-design gate

Once the research above is complete:

do not wait for another operator approval for this exact scope
create the corresponding design nodes immediately
reconcile them into the appropriate StarLang / starintel-server / auto-research design indexes
explicitly enumerate every required StarLang extension
separate language/compiler/runtime work from adapter/domain-server work
produce implementation slices/issues in dependency order
use StarLang first, then missing StarLang primitives, then Common Lisp adapters, with external processes only where technically justified

The design must show at least one end-to-end example for each:

static HTML scraper
JavaScript-rendered scraper
paginated JSON API scraper
RSS/Atom feed scraper
YaCy query adapter
SearX/SearXNG query adapter
local caller -> scraper -> reply
remote caller -> same scraper contract -> reply
expert/researcher requesting evidence through the scraping domain server

Completion criteria

Scrapy and the surrounding modern scraping/browser ecosystem are deeply enumerated rather than name-dropped.
Current StarLang implementation is checked before declaring gaps.
StarLang can define the scraper behavior; implementation-specific browser/network machinery stays behind adapters.
HTML, JS, APIs, RSS/Atom, YaCy and SearX/SearXNG are all covered.
Local and remote callers use one typed request/reply contract.
CAPTCHA continuation interoperates with #168.
Current StarIntel expert-system architecture is researched and integrated.
Current StarIntel agentic researcher/actor architecture is researched and integrated.
Security/resource/SSRF/browser concerns are explicit.
Research advances directly into design under this issue's operator approval.
No duplicated per-agent scraping stacks and no automatic Python-first fallback.

## Operator decision This scope is **pre-approved to advance from research directly into design** once the required research is complete and internally consistent. Do not stop for another approval gate between research and design unless the work would materially expand beyond this issue or discovers a real security/architecture blocker. Research **everything named below**. Do not reduce this to “port Scrapy.” The goal is to determine the StarLang/runtime semantics needed so StarIntel can define reusable scrapers and expose them through a first-class actor/domain-server contract. ## Goal Research and design a StarIntel scraping capability, implemented **StarLang-first**, that lets StarLang define scrapers for: - JavaScript-rendered pages - ordinary HTML pages - HTTP APIs - RSS / Atom feeds - YaCy instances as an example downstream integration - SearX / SearXNG instances as an example downstream integration The scraping actor/domain server must support **request/reply to either a local or remote caller through the same typed contract**. Locality is deployment/runtime topology, not a different API. The resulting design should be reusable by StarIntel expert systems, agents, Auto-Research, agentic researcher actors, OSINT domain servers, and ordinary StarIntel actors without each subsystem inventing its own HTTP/browser stack. ## Architecture direction Treat this as a coherent **Scraping / Source Acquisition Domain Server** or equivalent domain boundary after research validates the exact name/ownership. Do **not** create one actor per website or one actor per upstream scraping framework. Expected shape: ```text caller actor / expert / researcher -> typed scrape request -> local or remote Scraping Domain Server -> StarLang-defined scraper -> HTTP / browser / feed / API adapter -> typed extracted records + provenance -> reply / streamed observations / terminal outcome ``` Provider-specific/browser-specific implementation details stay behind capability adapters. ## 1. Deeply research Scrapy Research current Scrapy architecture and enumerate the reusable semantics StarIntel actually needs, including at minimum: - Spider definition model - start requests / request generation - Request / Response model - callbacks and errbacks - selectors and extraction - CSS/XPath semantics and selector composition - item/data models - item loaders where useful - pipelines / post-processing - downloader middleware - spider middleware - scheduler / request queue - duplicate filtering / request fingerprints - depth and traversal controls - redirects - cookies / sessions - authentication patterns - headers / user agents - retries / backoff - throttling / per-host concurrency - robots behavior and policy hooks - proxy support - DNS/network behavior relevant to scraping - HTTP caching - stateful/resumable jobs - persistence / job directories if still relevant - feed exports - signals / lifecycle hooks - extensions / plugin model - stats / instrumentation - error taxonomy - cancellation / shutdown behavior - async/concurrency architecture - browser/JavaScript integrations, especially the current Playwright ecosystem around Scrapy - API/JSON/XML extraction patterns - pagination/cursor traversal - sitemap/feed discovery where architecturally useful Do not cargo-cult Twisted, Python classes, or Scrapy internals into StarLang. Extract the semantic contract. ## 2. Research what StarLang must support to DEFINE scrapers Compare the requirements against the **current** `lost-rob0t/star-lang` compiler/runtime. For every item classify exactly one: 1. already supported 2. expressible as a StarLang library/pattern now 3. StarLang syntax/IR/runtime extension needed 4. Common Lisp adapter/runtime primitive needed below StarLang 5. isolated external process required Research at least these language/runtime semantics: ### Fetch / protocol - typed HTTP requests and responses - method, URL, query, headers, body - bounded response size - redirects - content-type dispatch - JSON, XML, HTML, text and binary/artifact refs - HTTP status/error classification - request deadlines - cancellation - bounded retry/backoff - per-host/global concurrency limits - rate limits and throttling - cache policy - proxy / transport selection - TLS policy - authenticated requests through **opaque credential/session refs**, never raw secrets in scraper source ### HTML extraction - DOM/document parse primitive - CSS selectors - XPath or a justified equivalent - attribute/text extraction - nested/repeating extraction - normalization/transforms - typed result schemas - required vs optional fields - extraction failure diagnostics ### JavaScript pages Research the browser-backed capability needed for: - navigate - execute page JavaScript when authorized - wait for DOM/network conditions - select/extract after rendering - click/fill/follow actions when a scraper requires interaction - frames if needed - pagination / infinite scroll patterns - network interception where useful - browser session/cookie state via opaque refs - bounded browser CPU/RAM/time - cancellation and stale-session fencing - screenshot/artifact refs only when requested Determine whether the correct implementation is a Common Lisp browser adapter, a Chrome/Playwright process adapter, another browser protocol adapter, or a combination. **StarLang remains the scraper definition/control language.** ### APIs Support reusable definitions for: - REST-ish JSON/XML APIs - query/body parameter templates - headers/auth refs - pagination (page, offset, cursor, Link header, custom token) - rate-limit observation - retry-after - schema extraction / normalization - typed API errors ### RSS / Atom Research first-class feed support or a small library pattern for: - RSS variants - Atom - feed metadata - entries/items - stable entry identity - publication/update timestamps - enclosures/links - incremental polling - dedup/change detection - provenance to the original feed and item ### Scraper composition Research a StarLang scraper-definition model for: - named scraper definitions - accepted input type - output schema/document type - steps / navigation / extraction - reusable extraction fragments - request templates - pagination/iteration - conditional branches - typed partial results - item emission during a run - terminal summary/result - provenance inheritance across request -> response -> extraction -> emitted document - deterministic scraper version identity - test fixtures / owned snapshots Do not invent a second general-purpose programming language inside StarLang. Prefer a small declarative surface plus ordinary StarLang expressions/actors where needed. ## 3. Local + remote actor request/reply contract Research and design one caller contract that works identically when the scraping actor/domain server is local or remote. Required semantics: - capability discovery - typed request and reply - correlation ID - caller identity / authorization - deadline propagation - cancellation propagation - idempotency identity where applicable - streamed/intermediate item observations where useful - final terminal outcome - typed `success | partial | blocked | unavailable | cancelled | failed`-style outcomes based on current StarLang conventions - bounded payloads - opaque artifact/session/result refs for large/sensitive state - preserved provenance / causal chain - retry semantics that do not accidentally duplicate effectful interactions - transparent routing through current StarLang domain-server/remoting semantics rather than separate local/remote implementations Cross-check existing approved StarLang research and current runtime rather than re-designing remoting from scratch. ## 4. CAPTCHA / challenge integration Integrate research with #168. A browser/HTTP scraper that encounters a challenge should be able to surface a typed challenge observation and optionally invoke `captcha.solve` through the capability/domain-server registry when policy permits. Do not hard-code CAPTCHA providers or private solver knowledge into scraper definitions. Preserve continuation state so the scrape can resume after challenge handling. ## 5. YaCy + SearX/SearXNG downstream examples Use YaCy and SearX/SearXNG as **example consumers/definitions**, not the architectural boundary. Research representative StarLang scraper/adapter definitions for: ### YaCy - query API/result extraction - pagination - endpoint capability/config - failure/rate/timeout behavior - provenance indicating the YaCy instance + underlying result URL/source ### SearX / SearXNG - machine-readable search endpoint(s) - engine/category parameters where supported - pagination - result normalization - provider/engine provenance when exposed - timeout/partial-result behavior These examples should prove the same scraper primitives can express a real API/metasearch integration without writing a bespoke runtime actor for each service. Coordinate conceptually with the StarIntel Search/provider work, but keep the scraping domain generic. ## 6. Research the StarIntel expert-system architecture Research the **current** StarIntel expert-system direction and implementation, including the approved StarLang expert-system research referenced by #156. Cover: - current StarLang expert-system language/runtime surface - expert-engine actor architecture - selectable logic backends (including Prolog where applicable) - fact/rule lifecycle - capability selection/delegation - explanation/proof traces - expert invocation from actors/domain servers - remote expert execution if supported/planned - how experts should select a scraper definition or scraping strategy - how scraped observations become typed facts/documents without bypassing provenance/validation - how scraper failures/partial results enter expert reasoning - how an expert asks for additional collection and receives the resulting evidence Do not revive a Prolog-only StarIntel actor architecture. StarLang owns the expert-engine boundary; Prolog may be a backend. ## 7. Research StarIntel agents + agentic researcher actors Research the current StarIntel actor/agent model and Auto-Research agentic-research architecture, including #45 and current implementation/research state. Enumerate: - ordinary StarIntel actors vs researcher/agent actors - research-run supervisors/control plane - worker/adaptor boundaries - tool/capability discovery - actor spawning/delegation if present - researcher planning and evidence acquisition - cancellation/budgets/deadlines - result/provenance handling - durable run state - local vs remote workers - browser-backed workers - how expert systems participate in planning/selection - how the new scraping domain server should be called by researchers Target architecture: - researcher/agent owns research intent/plan - expert system may decide/select collection strategy - scraping domain server owns source acquisition mechanics - scraper definition owns source-specific navigation/extraction - resulting observations/documents return through typed contracts with provenance Do **not** let every agent ship its own fetch/browser/scraping implementation. ## 8. Security / resource model Research and carry into design: - SSRF and private/metadata address policy - redirect-to-private-address handling - DNS rebinding considerations - browser sandboxing - untrusted HTML/JS - response/body size ceilings - download/artifact ceilings - CPU/RAM/browser page limits - per-host/global concurrency - rate-limit policy - credentials/session isolation - cookie isolation - cross-principal session leakage prevention - source allow/deny policy - audit/provenance without logging secrets/content unnecessarily - malicious feed/API payloads - decompression bombs - parser bombs / pathological DOMs - infinite pagination / crawl loops - visited-set / duplicate suppression - cancellation cleanup ## 9. Required research outputs Create/refresh research nodes covering at minimum: 1. Scrapy architecture and reusable semantics 2. StarLang scraper-definition language/runtime gaps 3. browser/JavaScript scraping adapter requirements 4. HTTP/API/HTML/feed acquisition semantics 5. local/remote scraping actor request/reply contract 6. YaCy/SearX example implementations 7. StarIntel expert-system integration 8. StarIntel agents / agentic researcher integration 9. adversarial security/resource analysis Update `roam/internal/OSINT-TOOLS-LEDGER.org` with Scrapy and any directly relevant browser/scraping technologies discovered during the pass. Cross-reference #165, #168, #156, and #45 where applicable. ## 10. Direct-to-design gate Once the research above is complete: - **do not wait for another operator approval** for this exact scope - create the corresponding design nodes immediately - reconcile them into the appropriate StarLang / starintel-server / auto-research design indexes - explicitly enumerate every required StarLang extension - separate language/compiler/runtime work from adapter/domain-server work - produce implementation slices/issues in dependency order - use StarLang first, then missing StarLang primitives, then Common Lisp adapters, with external processes only where technically justified The design must show at least one end-to-end example for each: - static HTML scraper - JavaScript-rendered scraper - paginated JSON API scraper - RSS/Atom feed scraper - YaCy query adapter - SearX/SearXNG query adapter - local caller -> scraper -> reply - remote caller -> same scraper contract -> reply - expert/researcher requesting evidence through the scraping domain server ## Completion criteria - Scrapy and the surrounding modern scraping/browser ecosystem are deeply enumerated rather than name-dropped. - Current StarLang implementation is checked before declaring gaps. - StarLang can define the scraper behavior; implementation-specific browser/network machinery stays behind adapters. - HTML, JS, APIs, RSS/Atom, YaCy and SearX/SearXNG are all covered. - Local and remote callers use one typed request/reply contract. - CAPTCHA continuation interoperates with #168. - Current StarIntel expert-system architecture is researched and integrated. - Current StarIntel agentic researcher/actor architecture is researched and integrated. - Security/resource/SSRF/browser concerns are explicit. - Research advances directly into design under this issue's operator approval. - No duplicated per-agent scraping stacks and no automatic Python-first fallback.

nsaspy referenced this issue

2026-08-29 10:28:16 +00:00

OSINT cluster research: username/account enumeration engines #180

nsaspy referenced this issue

2026-08-29 10:28:17 +00:00

StarLang research: OSINT domain-server orchestration gaps #175

nsaspy commented

2026-08-29 10:28:18 +00:00

Author

Owner

ARDR bounded research pass — scraper semantics / StarLang gap checkpoint

Selected as the initial concrete candidate from #170. This pass does not approve implementation.

Current-source verification

Current lost-rob0t/star-lang prototype/domain-server-core-prototype.lisp already provides a useful lower layer: compiled domain-server declarations with owned document types, accepted typed messages, declared tools, restart/mailbox/dispatcher/capabilities; typed domain tools with positive timeout-ms; a process-tool runner using TERM + kill-after; keyed domain-server instances; and typed handler dispatch. This is enough to avoid inventing a second actor framework for scraping.

The current domain-tool surface is intentionally much narrower than a scraper definition language: executable + argv template + input scalar + produced message + timeout + capabilities. It does not itself express HTTP request templates, response/content dispatch, DOM selectors, pagination, browser interactions, typed item emission, credential/session refs, crawl visited sets, or extraction pipelines. Those are real design gaps/patterns to classify rather than reasons to bypass StarLang.

Existing approved expert-system research (STAR-LANG-RESEARCH-023) reinforces an important boundary: expert rules describe knowledge and checked conclusions; arbitrary HTTP/process/actor effects are excluded from rule RHS. Therefore experts should request collection through the scraping/domain capability and reason over returned typed evidence, not embed scraper side effects inside inference rules.

Upstream evidence checkpoint

Scrapy 2.18's current docs separate semantics cleanly into scheduler/request queue, downloader middleware (HTTP-layer request/response behavior), spider middleware (response in / request+item out), item pipelines (validation/cleanup/dedup/storage), extensions/signals, and feed export. Its JOBDIR persistence persists scheduled requests, duplicate-filter state, and spider state so a crawl can resume. These are semantic responsibilities worth mapping; Twisted/Python classes are not.

SearXNG's current Search API is a strong downstream proof case: / and /search support GET/POST, format=json|csv|rss is instance-configurable, and queries expose q, categories, language, pageno, time range, safe search, etc. A disabled machine-readable format can produce 403, so the StarLang adapter must represent instance capability/configuration and typed unavailable/unsupported outcomes rather than assume JSON everywhere. SearXNG also has explicit engine suspension/backoff policy for access denied, CAPTCHA and 429 conditions, which supports the existing StarIntel direction of typed challenge/rate outcomes + provider health rather than blind retry.

YaCy remains a valid downstream instance boundary: current project docs describe local, organization portal, intranet, and P2P search modes, with JSON among its implementation technologies. A later pass still needs primary endpoint/schema verification before freezing the YaCy example contract.

StarLang classification from this pass

Already supported / reuse: domain-server ownership/accepted-message model; capability declaration; keyed actor/domain state; bounded process-tool timeout; runtime handler dispatch.

Library/pattern candidate (no syntax decision yet): typed scrape request/result messages; content-type dispatch; normalization pipelines; SearXNG query/result mapping; RSS/Atom mapping if ordinary typed iteration/composition is sufficient.

Likely StarLang surface/runtime extension requiring design: declarative request templates; selector/extraction expressions; bounded pagination/iteration + visited-set semantics; typed incremental item emission + terminal summary; opaque credential/session/artifact refs; provenance inheritance through fetch→extract→emit; optional capability/challenge continuation; scraper version identity/fixture binding.

Lower adapter primitive: HTTP client/DOM parser and browser protocol implementation should sit beneath StarLang rather than become language-specific provider actors.

External process only where justified: browser automation is the strongest candidate (Chromium/Playwright-style process/protocol boundary). StarLang must still own scraper definition/control semantics.

Design direction now sufficiently supported

Do not model ScrapyActor, SearXActor, etc. Model one Source Acquisition / Scraping Domain Server contract, with StarLang-defined source-specific scraper definitions and adapter capabilities beneath it. Local and remote callers must use the same typed operation; transport/locality stays below the contract.

Provisional flow:

researcher/actor -> typed acquire/scrape request -> domain server -> StarLang scraper definition -> HTTP|DOM|browser|feed adapter -> typed observations/items + provenance -> terminal outcome

Expert systems remain effect-closed: an expert can conclude that more evidence is needed / select a collection strategy, then the actor/researcher invokes the acquisition capability and returns evidence to inference.

Adversarial findings to carry into design

Do not expose arbitrary URL fetch as an unrestricted expert action: SSRF/confused-deputy risk.
Do not make browser sessions raw values in scraper source; use principal-scoped opaque refs with expiry/fencing.
Do not equate retry with safe replay after click/form interactions; effectful navigation needs idempotency/outcome semantics.
Pagination must be bounded and cycle-aware; otherwise a declarative scraper becomes an infinite crawler.
Machine-readable SearXNG formats are deployment capabilities, not guaranteed protocol facts.
Provider middleware/pipelines should not become a giant second programming language; keep a small scraper DSL plus normal StarLang composition.
Preserve raw response/artifact provenance separately from normalized records so extraction changes are auditable/replayable.

State

Research has materially advanced and the architecture direction is supported, but #169's full research scope is not complete yet (notably browser adapter primary-source pass, RSS/Atom semantics, YaCy endpoint/schema verification, full current StarLang remoting/cancellation/provenance cross-check, and agentic-researcher implementation inspection remain). Therefore this run does not mark the overall #169 research complete or DESIGN_READY_FOR_OPERATOR_REVIEW yet. No implementation issue was created and no implementation approval was inferred.

## ARDR bounded research pass — scraper semantics / StarLang gap checkpoint Selected as the initial concrete candidate from #170. This pass does **not** approve implementation. ### Current-source verification Current `lost-rob0t/star-lang` `prototype/domain-server-core-prototype.lisp` already provides a useful lower layer: compiled `domain-server` declarations with owned document types, accepted typed messages, declared tools, restart/mailbox/dispatcher/capabilities; typed domain tools with positive `timeout-ms`; a process-tool runner using TERM + kill-after; keyed domain-server instances; and typed handler dispatch. This is enough to avoid inventing a second actor framework for scraping. The current domain-tool surface is intentionally much narrower than a scraper definition language: executable + argv template + input scalar + produced message + timeout + capabilities. It does not itself express HTTP request templates, response/content dispatch, DOM selectors, pagination, browser interactions, typed item emission, credential/session refs, crawl visited sets, or extraction pipelines. Those are real design gaps/patterns to classify rather than reasons to bypass StarLang. Existing approved expert-system research (`STAR-LANG-RESEARCH-023`) reinforces an important boundary: expert rules describe knowledge and checked conclusions; arbitrary HTTP/process/actor effects are excluded from rule RHS. Therefore experts should request collection through the scraping/domain capability and reason over returned typed evidence, not embed scraper side effects inside inference rules. ### Upstream evidence checkpoint Scrapy 2.18's current docs separate semantics cleanly into scheduler/request queue, downloader middleware (HTTP-layer request/response behavior), spider middleware (response in / request+item out), item pipelines (validation/cleanup/dedup/storage), extensions/signals, and feed export. Its JOBDIR persistence persists scheduled requests, duplicate-filter state, and spider state so a crawl can resume. These are semantic responsibilities worth mapping; Twisted/Python classes are not. SearXNG's current Search API is a strong downstream proof case: `/` and `/search` support GET/POST, `format=json|csv|rss` is instance-configurable, and queries expose `q`, categories, language, `pageno`, time range, safe search, etc. A disabled machine-readable format can produce 403, so the StarLang adapter must represent instance capability/configuration and typed unavailable/unsupported outcomes rather than assume JSON everywhere. SearXNG also has explicit engine suspension/backoff policy for access denied, CAPTCHA and 429 conditions, which supports the existing StarIntel direction of typed challenge/rate outcomes + provider health rather than blind retry. YaCy remains a valid downstream instance boundary: current project docs describe local, organization portal, intranet, and P2P search modes, with JSON among its implementation technologies. A later pass still needs primary endpoint/schema verification before freezing the YaCy example contract. ### StarLang classification from this pass **Already supported / reuse:** domain-server ownership/accepted-message model; capability declaration; keyed actor/domain state; bounded process-tool timeout; runtime handler dispatch. **Library/pattern candidate (no syntax decision yet):** typed scrape request/result messages; content-type dispatch; normalization pipelines; SearXNG query/result mapping; RSS/Atom mapping if ordinary typed iteration/composition is sufficient. **Likely StarLang surface/runtime extension requiring design:** declarative request templates; selector/extraction expressions; bounded pagination/iteration + visited-set semantics; typed incremental item emission + terminal summary; opaque credential/session/artifact refs; provenance inheritance through fetch→extract→emit; optional capability/challenge continuation; scraper version identity/fixture binding. **Lower adapter primitive:** HTTP client/DOM parser and browser protocol implementation should sit beneath StarLang rather than become language-specific provider actors. **External process only where justified:** browser automation is the strongest candidate (Chromium/Playwright-style process/protocol boundary). StarLang must still own scraper definition/control semantics. ### Design direction now sufficiently supported Do not model `ScrapyActor`, `SearXActor`, etc. Model one Source Acquisition / Scraping Domain Server contract, with StarLang-defined source-specific scraper definitions and adapter capabilities beneath it. Local and remote callers must use the same typed operation; transport/locality stays below the contract. Provisional flow: `researcher/actor -> typed acquire/scrape request -> domain server -> StarLang scraper definition -> HTTP|DOM|browser|feed adapter -> typed observations/items + provenance -> terminal outcome` Expert systems remain effect-closed: an expert can conclude that more evidence is needed / select a collection strategy, then the actor/researcher invokes the acquisition capability and returns evidence to inference. ### Adversarial findings to carry into design 1. Do not expose arbitrary URL fetch as an unrestricted expert action: SSRF/confused-deputy risk. 2. Do not make browser sessions raw values in scraper source; use principal-scoped opaque refs with expiry/fencing. 3. Do not equate retry with safe replay after click/form interactions; effectful navigation needs idempotency/outcome semantics. 4. Pagination must be bounded and cycle-aware; otherwise a declarative scraper becomes an infinite crawler. 5. Machine-readable SearXNG formats are deployment capabilities, not guaranteed protocol facts. 6. Provider middleware/pipelines should not become a giant second programming language; keep a small scraper DSL plus normal StarLang composition. 7. Preserve raw response/artifact provenance separately from normalized records so extraction changes are auditable/replayable. ### State Research has materially advanced and the architecture direction is supported, but #169's full research scope is not complete yet (notably browser adapter primary-source pass, RSS/Atom semantics, YaCy endpoint/schema verification, full current StarLang remoting/cancellation/provenance cross-check, and agentic-researcher implementation inspection remain). Therefore this run **does not mark the overall #169 research complete or DESIGN_READY_FOR_OPERATOR_REVIEW yet**. No implementation issue was created and no implementation approval was inferred.

nsaspy commented

2026-08-29 10:28:18 +00:00

Author

Owner

ARDR bounded research completion — READY_FOR_DESIGN → design produced

This cycle completes the remaining research gaps identified in the previous checkpoint and advances the scoped work to design under the operator's existing research→design authorization. This does not approve implementation.

Design produced: #171 — DESIGN_READY_FOR_OPERATOR_REVIEW.

First bounded implementation proposal produced: lost-rob0t/star-lang#50, explicitly AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL.

Current StarLang remoting / cancellation / durability verification

Current starlang-runtime/src/wire-dispatcher.lisp proves the lifecycle substrate needed by acquisition:

correlation/idempotency identity includes actor, sender, correlation ID, idempotency key, reply target, deadline and payload;
terminal outcomes replay without re-running completed work;
cancellation targets either message ID or correlation ID and turns active work terminal with star.cancelled;
expired deadlines produce terminal star.deadline-exceeded;
deferred completion cannot silently re-defer the same command.

Current prototype/bbp-domain-remoting-tests.lisp proves the same domain lifecycle can execute through a remote node: main gateway accepts the command, remote execution returns reply+completion, pending state clears, duplicate delivery replays terminal outcomes without re-running the tool, and remote nodes heartbeat/register through a remoting port. Therefore acquisition locality is a routing/runtime concern, not a second source API.

Current star-journal/src/journal.lisp already defines runtime journal events pending, route-result, and remote-result, validates lifecycle command envelopes, and restricts settled dispatch outcomes to complete/retry/fail. Persisted scraper/crawl state should extend/reuse this ownership rather than invent a second scheduler journal.

Browser / JavaScript primary-source findings

Current Playwright documentation confirms the adapter behaviors needed beneath StarLang:

BrowserContext provides isolated cookie/local/session-storage environments and cheap independent contexts;
pages support navigation, fill/click and popup/tab workflows;
navigation distinguishes commit/loading stages and actions auto-wait for actionable state;
context/page network routing can observe/modify/abort HTTP(S) traffic, including XHR/fetch;
service workers complicate network interception and must be handled as an explicit capability/diagnostic rather than assuming every request is visible to ordinary route handlers.

Design consequence: browser execution belongs behind a bounded browser capability adapter with principal-scoped opaque context/session refs. StarLang owns navigation/extraction intent; native Playwright objects never enter StarLang IR.

Primary references retrieved 2026-08-29:

RSS / Atom semantics

RSS 2.0 is XML with channel metadata and item records; item guid, links, publication date, enclosure and namespaced extensions are useful but not all feeds provide one universally perfect identity key. The adapter therefore needs deterministic fallback identity with provenance when a stable GUID is absent.

Atom RFC 4287 gives a stronger identity/update contract:

feed and entry each require exactly one atom:id;
feed requires atom:updated;
entry may carry published and must represent update time independently;
repeated entries with the same atom:id identify the same entry, with latest updated being a normal processor choice;
links are typed by relation such as self/alternate.

Design consequence: feed handling is likely a typed adapter/library pattern over HTTP + XML + bounded iteration, not necessarily new dedicated syntax. Preserve published vs updated, original feed identity, item/entry identity, extension namespaces and capture provenance.

Primary references retrieved 2026-08-29:

YaCy verification

The current YaCy search-server repository (updated July 2026) states that its APIs are HTTP/XML and HTTP/JSON and that customized web applications can attach through those interfaces. The documented search interface exposes yacysearch.json; older but project-owned API documentation gives the concrete /yacysearch.json?query=...&resource=... pattern. Current related YaCy projects continue to use /yacysearch.json as a compatibility path.

Important negative finding: deep record offsets have historically had practical limits. Therefore the StarLang YaCy source definition must capability-test/bound pagination rather than assume arbitrary startRecord depth.

Primary/current references retrieved 2026-08-29:

Agentic researcher boundary verified

Current canonical Auto-Research STAR-RESEARCH-002 Deep Research Execution Adapter Analysis already defines:

Research Run Supervisor owns the run;
Branch Supervisor owns candidate isolation and budgets;
append-only ledgers own durable research state;
deep-research engines/browser workers are bounded adapters;
workers may retrieve/summarize/return normalized events but cannot promote architecture, mutate canonical run records, or own canonical Org state;
browser JavaScript actors are optional worker runtimes behind the same adapter envelopes.

Design consequence: Source Acquisition sits beneath researcher adapters as the reusable collection mechanism. It does not replace ADARD/ARDR supervision or become a second research control plane.

Expert-system boundary retained

Existing StarLang expert research requires declarative/effect-closed rule actions: HTTP, process launch and arbitrary actor sends are disallowed inside rule RHS. An expert may derive evidence-needed or select a collection strategy; the owning actor/researcher invokes Source Acquisition and feeds returned typed evidence back into the expert session. partial/blocked acquisition must never collapse into logical absence/not-found.

Final research classification

Already supported: domain-server ownership, typed lifecycle envelopes, correlation, deadline, cancellation, idempotent terminal replay, deferred completion, remote round-trip, bounded mailboxes, runtime journaling.

StarLang extension: scraper/source declaration + closed acquisition IR; deterministic version/fixture identity; bounded page/offset/cursor traversal + visited set; incremental bounded item emission; provenance inheritance; opaque scoped refs; optional challenge/capability continuation where not provided by shared work.

Common Lisp adapter beneath StarLang: HTTP transport, DOM/XML parsing, feed normalization and provider-independent request execution.

Isolated external process justified: browser automation, with Chromium/Playwright-style adapter as the strongest current candidate; StarLang remains definition/control authority.

Adversarial closeout

The research rejects:

Scrapy/Twisted as StarIntel architecture;
one actor per source;
arbitrary URL/browser actions directly from expert rules;
browser workers owning research state;
separate local/remote scraper APIs;
generic retry/replay for clicks/forms;
unbounded pagination/fan-out;
guaranteed SearXNG JSON or arbitrarily deep YaCy pagination;
raw browser/cookie/credential state in StarLang source.

No unresolved finding now requires changing the proposed Source Acquisition / Scraping Domain Server boundary.

State

Research: READY_FOR_DESIGN (completed for #169 scope)
Design: produced as #171, DESIGN_READY_FOR_OPERATOR_REVIEW
Proposed implementation: lost-rob0t/star-lang#50
Implementation approval: PENDING / OPERATOR ONLY

No product code was implemented and nothing was marked approved for implementation.

## ARDR bounded research completion — READY_FOR_DESIGN → design produced This cycle completes the remaining research gaps identified in the previous checkpoint and advances the scoped work to design under the operator's existing research→design authorization. **This does not approve implementation.** Design produced: #171 — `DESIGN_READY_FOR_OPERATOR_REVIEW`. First bounded implementation proposal produced: `lost-rob0t/star-lang#50`, explicitly `AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL`. ### Current StarLang remoting / cancellation / durability verification Current `starlang-runtime/src/wire-dispatcher.lisp` proves the lifecycle substrate needed by acquisition: - correlation/idempotency identity includes actor, sender, correlation ID, idempotency key, reply target, deadline and payload; - terminal outcomes replay without re-running completed work; - cancellation targets either message ID or correlation ID and turns active work terminal with `star.cancelled`; - expired deadlines produce terminal `star.deadline-exceeded`; - deferred completion cannot silently re-defer the same command. Current `prototype/bbp-domain-remoting-tests.lisp` proves the same domain lifecycle can execute through a remote node: main gateway accepts the command, remote execution returns reply+completion, pending state clears, duplicate delivery replays terminal outcomes without re-running the tool, and remote nodes heartbeat/register through a remoting port. Therefore acquisition locality is a routing/runtime concern, not a second source API. Current `star-journal/src/journal.lisp` already defines runtime journal events `pending`, `route-result`, and `remote-result`, validates lifecycle command envelopes, and restricts settled dispatch outcomes to complete/retry/fail. Persisted scraper/crawl state should extend/reuse this ownership rather than invent a second scheduler journal. ### Browser / JavaScript primary-source findings Current Playwright documentation confirms the adapter behaviors needed beneath StarLang: - `BrowserContext` provides isolated cookie/local/session-storage environments and cheap independent contexts; - pages support navigation, fill/click and popup/tab workflows; - navigation distinguishes commit/loading stages and actions auto-wait for actionable state; - context/page network routing can observe/modify/abort HTTP(S) traffic, including XHR/fetch; - service workers complicate network interception and must be handled as an explicit capability/diagnostic rather than assuming every request is visible to ordinary route handlers. Design consequence: browser execution belongs behind a bounded browser capability adapter with principal-scoped opaque context/session refs. StarLang owns navigation/extraction intent; native Playwright objects never enter StarLang IR. Primary references retrieved 2026-08-29: - https://playwright.dev/docs/browser-contexts - https://playwright.dev/docs/pages - https://playwright.dev/docs/network - https://playwright.dev/docs/api/class-browsercontext ### RSS / Atom semantics RSS 2.0 is XML with channel metadata and item records; item `guid`, links, publication date, enclosure and namespaced extensions are useful but not all feeds provide one universally perfect identity key. The adapter therefore needs deterministic fallback identity with provenance when a stable GUID is absent. Atom RFC 4287 gives a stronger identity/update contract: - feed and entry each require exactly one `atom:id`; - feed requires `atom:updated`; - entry may carry `published` and must represent update time independently; - repeated entries with the same `atom:id` identify the same entry, with latest `updated` being a normal processor choice; - links are typed by relation such as `self`/`alternate`. Design consequence: feed handling is likely a typed adapter/library pattern over HTTP + XML + bounded iteration, not necessarily new dedicated syntax. Preserve `published` vs `updated`, original feed identity, item/entry identity, extension namespaces and capture provenance. Primary references retrieved 2026-08-29: - https://www.rssboard.org/rss-specification - https://www.rfc-editor.org/info/rfc4287/ ### YaCy verification The current YaCy search-server repository (updated July 2026) states that its APIs are HTTP/XML and HTTP/JSON and that customized web applications can attach through those interfaces. The documented search interface exposes `yacysearch.json`; older but project-owned API documentation gives the concrete `/yacysearch.json?query=...&resource=...` pattern. Current related YaCy projects continue to use `/yacysearch.json` as a compatibility path. Important negative finding: deep record offsets have historically had practical limits. Therefore the StarLang YaCy source definition must capability-test/bound pagination rather than assume arbitrary `startRecord` depth. Primary/current references retrieved 2026-08-29: - https://github.com/yacy/yacy_search_server - https://wiki.yacy.net/index.php/Dev%3AAPIyacysearch - https://github.com/yacy/yacy_expert ### Agentic researcher boundary verified Current canonical Auto-Research `STAR-RESEARCH-002 Deep Research Execution Adapter Analysis` already defines: - Research Run Supervisor owns the run; - Branch Supervisor owns candidate isolation and budgets; - append-only ledgers own durable research state; - deep-research engines/browser workers are bounded adapters; - workers may retrieve/summarize/return normalized events but cannot promote architecture, mutate canonical run records, or own canonical Org state; - browser JavaScript actors are optional worker runtimes behind the same adapter envelopes. Design consequence: Source Acquisition sits beneath researcher adapters as the reusable collection mechanism. It does not replace ADARD/ARDR supervision or become a second research control plane. ### Expert-system boundary retained Existing StarLang expert research requires declarative/effect-closed rule actions: HTTP, process launch and arbitrary actor sends are disallowed inside rule RHS. An expert may derive `evidence-needed` or select a collection strategy; the owning actor/researcher invokes Source Acquisition and feeds returned typed evidence back into the expert session. `partial`/`blocked` acquisition must never collapse into logical absence/not-found. ### Final research classification **Already supported:** domain-server ownership, typed lifecycle envelopes, correlation, deadline, cancellation, idempotent terminal replay, deferred completion, remote round-trip, bounded mailboxes, runtime journaling. **StarLang extension:** scraper/source declaration + closed acquisition IR; deterministic version/fixture identity; bounded page/offset/cursor traversal + visited set; incremental bounded item emission; provenance inheritance; opaque scoped refs; optional challenge/capability continuation where not provided by shared work. **Common Lisp adapter beneath StarLang:** HTTP transport, DOM/XML parsing, feed normalization and provider-independent request execution. **Isolated external process justified:** browser automation, with Chromium/Playwright-style adapter as the strongest current candidate; StarLang remains definition/control authority. ### Adversarial closeout The research rejects: - Scrapy/Twisted as StarIntel architecture; - one actor per source; - arbitrary URL/browser actions directly from expert rules; - browser workers owning research state; - separate local/remote scraper APIs; - generic retry/replay for clicks/forms; - unbounded pagination/fan-out; - guaranteed SearXNG JSON or arbitrarily deep YaCy pagination; - raw browser/cookie/credential state in StarLang source. No unresolved finding now requires changing the proposed Source Acquisition / Scraping Domain Server boundary. ### State - Research: `READY_FOR_DESIGN` (completed for #169 scope) - Design: produced as #171, `DESIGN_READY_FOR_OPERATOR_REVIEW` - Proposed implementation: `lost-rob0t/star-lang#50` - Implementation approval: **PENDING / OPERATOR ONLY** No product code was implemented and nothing was marked approved for implementation.

nsaspy referenced this issue

2026-09-02 17:26:15 +00:00

ARADR: rework StarIntel inference into a StarLang expert-engine actor #182