Add StarIntel scrapers and RabbitMQ spider tutorial #6

Open

lost-rob0t wants to merge 3 commits from agent/starintel-scrapers-spiders-tutorial into main

lost-rob0t commented

2026-08-13 06:36:58 +00:00

(Migrated from github.com)

What changed

adds tutorials/00-starintel-server-scrapers-spiders.org, a 4,447-line code-first tutorial with 195 numbered teaching sections
adds projects/starintel-server.org as the canonical Org-roam project node
indexes both nodes in index.org

Project A — StarIntel Server

The tutorial gives a short reproducible bootstrap for the real lost-rob0t/starintel-server stack using Nix + Docker Compose, then verifies HTTP health, RabbitMQ, CouchDB, first-run authentication, and the exact distributed document topology.

It is source-verified against StarIntel Server master commit e2fa1991d8b08b74bcb2194c7a10f02da2431ad1 and teaches the actual current messaging contract:

durable topic exchange documents
ingest queue/binding documents.ingest / documents.ingest.#
producer keys documents.ingest.<dtype>
post-insert events documents.new.<dtype> after CouchDB _id / _rev
remote target routing actors.<actor>.new.target
at-least-once delivery and required idempotency

Project B — scraper/spider

The tutorial starts from a one-page requests/BeautifulSoup scraper and grows it into a Scrapy project backed by a deterministic local fixture website. It covers:

crawl frontier and link cycles
canonical URLs and normalized content
deterministic snapshot IDs
StarIntel URL bridge documents
RabbitMQ publisher confirms and persistent messages
CouchDB verification and live documents.new.url observation
Scrapy item pipelines
relation emission (links-to)
target-driven external actor mode using actors.web-spider.targets
manual ACK/NACK semantics
bounded retries / dead-letter design
crawl depth, hop, byte, page, and wall-time budgets
robots/rate limiting/429 behavior
SSRF and redirect scope controls
raw archive/versioned parser design
unit, spider, Rabbit, restart, and full E2E tests
horizontal scaling and distributed per-domain rate limiting
explicit migration boundary from the current flat server bridge toward strict StarIntel 0.9

Source grounding

Examples were checked against current starintel-server implementation and docs, especially:

docs/messaging.org
docs/document-spec.org
docs/configuration.org
docs/actors.org
docker-compose.yml
scripts/stack-test.sh
source/rabbit.lisp

The tutorial explicitly distinguishes current permissive Rabbit/HTTP ingest behavior from strict 0.9 schema validation rather than describing planned/final behavior as already enforced.

Validation

Org source blocks/examples/quotes were checked for balanced delimiters
tutorial uses the incoming numbered NN-<project>-<slug>.org convention
stable Org-roam IDs are used for the tutorial and project node
branch is isolated from the Prolog-RLM/EPUB work

Merge only after repository CI is green.

## What changed - adds `tutorials/00-starintel-server-scrapers-spiders.org`, a 4,447-line code-first tutorial with 195 numbered teaching sections - adds `projects/starintel-server.org` as the canonical Org-roam project node - indexes both nodes in `index.org` ## Project A — StarIntel Server The tutorial gives a short reproducible bootstrap for the real `lost-rob0t/starintel-server` stack using Nix + Docker Compose, then verifies HTTP health, RabbitMQ, CouchDB, first-run authentication, and the exact distributed document topology. It is source-verified against StarIntel Server `master` commit `e2fa1991d8b08b74bcb2194c7a10f02da2431ad1` and teaches the actual current messaging contract: - durable topic exchange `documents` - ingest queue/binding `documents.ingest` / `documents.ingest.#` - producer keys `documents.ingest.<dtype>` - post-insert events `documents.new.<dtype>` after CouchDB `_id` / `_rev` - remote target routing `actors.<actor>.new.target` - at-least-once delivery and required idempotency ## Project B — scraper/spider The tutorial starts from a one-page `requests`/BeautifulSoup scraper and grows it into a Scrapy project backed by a deterministic local fixture website. It covers: - crawl frontier and link cycles - canonical URLs and normalized content - deterministic snapshot IDs - StarIntel URL bridge documents - RabbitMQ publisher confirms and persistent messages - CouchDB verification and live `documents.new.url` observation - Scrapy item pipelines - relation emission (`links-to`) - target-driven external actor mode using `actors.web-spider.targets` - manual ACK/NACK semantics - bounded retries / dead-letter design - crawl depth, hop, byte, page, and wall-time budgets - robots/rate limiting/429 behavior - SSRF and redirect scope controls - raw archive/versioned parser design - unit, spider, Rabbit, restart, and full E2E tests - horizontal scaling and distributed per-domain rate limiting - explicit migration boundary from the current flat server bridge toward strict StarIntel 0.9 ## Source grounding Examples were checked against current `starintel-server` implementation and docs, especially: - `docs/messaging.org` - `docs/document-spec.org` - `docs/configuration.org` - `docs/actors.org` - `docker-compose.yml` - `scripts/stack-test.sh` - `source/rabbit.lisp` The tutorial explicitly distinguishes current permissive Rabbit/HTTP ingest behavior from strict 0.9 schema validation rather than describing planned/final behavior as already enforced. ## Validation - Org source blocks/examples/quotes were checked for balanced delimiters - tutorial uses the incoming numbered `NN-<project>-<slug>.org` convention - stable Org-roam IDs are used for the tutorial and project node - branch is isolated from the Prolog-RLM/EPUB work Merge only after repository CI is green.

nsaspy changed title from ~~WIP: Add StarIntel scrapers and RabbitMQ spider tutorial~~ to Add StarIntel scrapers and RabbitMQ spider tutorial

2026-08-27 19:42:16 +00:00

This pull request has changes conflicting with the target branch.

index.org

View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Merge commit title

Merge pull request 'Add StarIntel scrapers and RabbitMQ spider tutorial' (#6) from agent/starintel-scrapers-spiders-tutorial into main

Merge commit body

Reviewed-on: https://git.starintel.actor/nsaspy/gpt-org-roam/pulls/6

Merge pull request 'Add StarIntel scrapers and RabbitMQ spider tutorial' (#6) from agent/starintel-scrapers-spiders-tutorial into main

Reviewed-on: https://git.starintel.actor/nsaspy/gpt-org-roam/pulls/6

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin agent/starintel-scrapers-spiders-tutorial:agent/starintel-scrapers-spiders-tutorial

git switch agent/starintel-scrapers-spiders-tutorial

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff agent/starintel-scrapers-spiders-tutorial

git switch agent/starintel-scrapers-spiders-tutorial

git rebase main

git switch main

git merge --ff-only agent/starintel-scrapers-spiders-tutorial

git switch agent/starintel-scrapers-spiders-tutorial

git rebase main

git switch main

git merge --no-ff agent/starintel-scrapers-spiders-tutorial

git switch main

git merge --squash agent/starintel-scrapers-spiders-tutorial