Add actor-native scraping library core #24

Merged

lost-rob0t merged 14 commits from agent/scraping-library-core into main

2026-08-13 06:39:59 +00:00

lost-rob0t commented

2026-08-13 06:37:26 +00:00

(Migrated from github.com)

What changed

This implements the first StarLang scraping-library slice on top of the final actor runtime.

`star-http-port`

replaces the placeholder package with an injectable HTTP client/request/response API
validates methods, timeouts, and redirect limits
keeps deterministic tests network-free
adds make-dexador-http-client, which dynamically loads Dexador only when requested instead of making the base StarLang runtime depend on a network stack
maps Dexador response body/status/headers/final URI into StarLang HTTP response objects

`star-scrape`

Adds a final ASDF system with:

injectable HTML adapter interface
declarative scrape fields and scrape plans
text and attribute extraction
one-or-many field cardinality
required-field validation
HTTP failure handling
scrape result provenance for requested/final URL, status, response headers, and named extracted fields
optional make-plump-clss-html-adapter that dynamically loads lQuery/Plump/CLSS and probes current CLSS:QUERY with compatibility fallback to older CLSS:SELECT

Actor integration

Scraping is a normal StarLang actor capability:

make-scraper-actor-definition
create-scraper-actor

The acceptance test creates star://scrape:localhost:page-scraper, registers it in starlang-runtime, invokes a scrape plan through the actor URI, and verifies the extracted result.

Validation

star-http-port tests no longer depend on FiveAM
star-scrape tests use injected HTTP/DOM fixtures, never live network
star-scrape added to ci/target-systems.txt
CI directly tests star-http-port + star-scrape and asserts they do not load starlang-prototype
Nix checks and installed starlang-test runner execute the same final-system tests

Scope / follow-on

This slice deliberately keeps Dexador and Plump/CLSS optional at load time. Once their exact Nix package closure is pinned, a follow-up can add live backend smoke tests plus robots/rate-limit/retry policy without weakening deterministic base CI.

## What changed This implements the first StarLang scraping-library slice on top of the final actor runtime. ### `star-http-port` - replaces the placeholder package with an injectable HTTP client/request/response API - validates methods, timeouts, and redirect limits - keeps deterministic tests network-free - adds `make-dexador-http-client`, which dynamically loads Dexador only when requested instead of making the base StarLang runtime depend on a network stack - maps Dexador response body/status/headers/final URI into StarLang HTTP response objects ### `star-scrape` Adds a final ASDF system with: - injectable HTML adapter interface - declarative scrape fields and scrape plans - text and attribute extraction - one-or-many field cardinality - required-field validation - HTTP failure handling - scrape result provenance for requested/final URL, status, response headers, and named extracted fields - optional `make-plump-clss-html-adapter` that dynamically loads lQuery/Plump/CLSS and probes current `CLSS:QUERY` with compatibility fallback to older `CLSS:SELECT` ### Actor integration Scraping is a normal StarLang actor capability: - `make-scraper-actor-definition` - `create-scraper-actor` The acceptance test creates `star://scrape:localhost:page-scraper`, registers it in `starlang-runtime`, invokes a scrape plan through the actor URI, and verifies the extracted result. ### Validation - `star-http-port` tests no longer depend on FiveAM - `star-scrape` tests use injected HTTP/DOM fixtures, never live network - `star-scrape` added to `ci/target-systems.txt` - CI directly tests `star-http-port` + `star-scrape` and asserts they do not load `starlang-prototype` - Nix checks and installed `starlang-test` runner execute the same final-system tests ## Scope / follow-on This slice deliberately keeps Dexador and Plump/CLSS optional at load time. Once their exact Nix package closure is pinned, a follow-up can add live backend smoke tests plus robots/rate-limit/retry policy without weakening deterministic base CI.