PROPOSED: StarLang scraper declaration + typed acquisition IR [AWAITING OPERATOR APPROVAL] #51

New issue

Open

opened 2026-08-29 10:27:55 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-08-29 10:27:55 +00:00

Owner

State

AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL

This issue is a proposed implementation slice derived from lost-rob0t/starintel-auto-research#169 and design lost-rob0t/starintel-auto-research#171.

Do not implement or expose to the executable RAGE queue until the operator explicitly approves implementation. Research/design readiness is not implementation approval.

Priority / dependency role

Priority class: unlabelled dependency-unblocking foundational slice.
Dependency role: prerequisite for bounded pagination/traversal, adapter implementations, Source Acquisition Domain Server integration, and downstream SearXNG/YaCy scraper definitions.
Existing StarLang domain-server/remoting/lifecycle runtime must be reused; no second actor or transport framework.

Bounded scope

Add the smallest closed StarLang compiler/IR surface needed to represent a scraper/source definition without implementing HTTP, DOM, browser, feed, YaCy, or SearX adapters yet.

Required contract:

new closed declaration kind for a source/scraper definition;
stable scraper name + semantic/version identity;
declared accepted input type;
declared emitted schema/message/document types;
declared capability refs such as http, dom, browser, feed as inert typed requirements, not host objects;
closed typed acquisition request/response/item/terminal IR structures;
deterministic fixture/source-definition identity suitable for conformance tests;
explicit provenance linkage fields from acquisition request through source response/artifact to emitted item;
portable manifest projection for the new declaration where appropriate.

Non-goals

no HTTP client
no DOM parser
no browser process
no feed parser
no pagination/visited-set implementation
no CAPTCHA continuation
no starintel-server runtime integration
no SearXNG/YaCy source implementation
no arbitrary host calls or shell escape from scraper source

Mandatory RED-first target

Before production mutation, add a valid StarLang fixture containing the smallest legal scraper declaration expected by design #171.

Run it against untouched current main and record RED evidence showing the parser/compiler rejects or cannot compile the declaration because the scraper declaration/typed acquisition IR does not exist.

Also add negative fixtures that must remain rejected after implementation:

scraper definition naming an arbitrary host function/process;
raw credential/token literal in a credential/session field where an opaque ref is required;
undeclared output type/capability;
malformed/unversioned scraper identity.

The RED failure must be due to the missing language feature, not a broken test harness.

Acceptance criteria

valid scraper declaration parses/compiles into closed typed IR;
exact declaration/version/fixture identity is deterministic;
input/output schema references are compile-time validated;
capability refs are closed/typed and contain no runtime implementation objects;
provenance fields required by design #171 survive portable IR/manifest projection;
invalid arbitrary-host/raw-secret/unknown-schema fixtures fail deterministically;
existing actor/domain-server fixtures and canonical manifests remain compatible or receive an explicit versioned migration;
repository-native deterministic tests are green on exact head.

Authority

Design: lost-rob0t/starintel-auto-research#171

Implementation approval: PENDING / operator only.

## State `AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL` This issue is a proposed implementation slice derived from `lost-rob0t/starintel-auto-research#169` and design `lost-rob0t/starintel-auto-research#171`. **Do not implement or expose to the executable RAGE queue until the operator explicitly approves implementation.** Research/design readiness is not implementation approval. ## Priority / dependency role - Priority class: unlabelled dependency-unblocking foundational slice. - Dependency role: prerequisite for bounded pagination/traversal, adapter implementations, Source Acquisition Domain Server integration, and downstream SearXNG/YaCy scraper definitions. - Existing StarLang domain-server/remoting/lifecycle runtime must be reused; no second actor or transport framework. ## Bounded scope Add the smallest closed StarLang compiler/IR surface needed to represent a scraper/source definition without implementing HTTP, DOM, browser, feed, YaCy, or SearX adapters yet. Required contract: - new closed declaration kind for a source/scraper definition; - stable scraper name + semantic/version identity; - declared accepted input type; - declared emitted schema/message/document types; - declared capability refs such as `http`, `dom`, `browser`, `feed` as inert typed requirements, not host objects; - closed typed acquisition request/response/item/terminal IR structures; - deterministic fixture/source-definition identity suitable for conformance tests; - explicit provenance linkage fields from acquisition request through source response/artifact to emitted item; - portable manifest projection for the new declaration where appropriate. ## Non-goals - no HTTP client - no DOM parser - no browser process - no feed parser - no pagination/visited-set implementation - no CAPTCHA continuation - no starintel-server runtime integration - no SearXNG/YaCy source implementation - no arbitrary host calls or shell escape from scraper source ## Mandatory RED-first target Before production mutation, add a valid StarLang fixture containing the smallest legal scraper declaration expected by design #171. Run it against untouched current main and record RED evidence showing the parser/compiler rejects or cannot compile the declaration because the scraper declaration/typed acquisition IR does not exist. Also add negative fixtures that must remain rejected after implementation: - scraper definition naming an arbitrary host function/process; - raw credential/token literal in a credential/session field where an opaque ref is required; - undeclared output type/capability; - malformed/unversioned scraper identity. The RED failure must be due to the missing language feature, not a broken test harness. ## Acceptance criteria - valid scraper declaration parses/compiles into closed typed IR; - exact declaration/version/fixture identity is deterministic; - input/output schema references are compile-time validated; - capability refs are closed/typed and contain no runtime implementation objects; - provenance fields required by design #171 survive portable IR/manifest projection; - invalid arbitrary-host/raw-secret/unknown-schema fixtures fail deterministically; - existing actor/domain-server fixtures and canonical manifests remain compatible or receive an explicit versioned migration; - repository-native deterministic tests are green on exact head. ## Authority Design: `lost-rob0t/starintel-auto-research#171` Implementation approval: **PENDING / operator only**.

nsaspy referenced this issue

2026-09-02 17:26:05 +00:00

Browser evidence actor system for bounded screenshot/DOM acquisition #56