nsaspy/starintel-server
1
0
Fork 0
Code Issues 49 Pull requests 7 Projects Releases Packages Wiki Activity Actions

Add Docker-first modern website actor system #145

Open
nsaspy wants to merge 6 commits from feat/modern-website-actors into master
pull from: feat/modern-website-actors
merge into: nsaspy:master
Conversation 1 Commits 6 Files changed 32 +1350
nsaspy commented 2026-09-08 01:54:08 +00:00
Owner
Copy link

Summary

Moves the archived WEF / Young Global Leaders scraper port into the normal/public StarIntel actor home (starintel-server) and replaces the old Scrapy/shell deployment model with an extensible modern Python actor runtime plus an independent Docker Compose actor plane.

The mistakenly-created starintel-pro-actors PR #14 is closed and superseded.

Actor runtime

  • actors/website/ is a standalone Python 3.12 actor package
  • asyncio mailbox actor model with bounded crawl scheduling
  • generic SiteAdapter, fetcher, registry, sink, request/batch/report contracts
  • third-party site plugins can register through starintel.website_actors entry points
  • HTTP-first httpx; Playwright Chromium fallback for JS shells, 401/403/429, and HTTP failures
  • one browser process per crawl with isolated browser contexts
  • images/media/fonts blocked for browser collection efficiency
  • deterministic StarIntel v0.9 IDs and provenance
  • JSONL standalone/debug sink and durable RabbitMQ ingest sink

Current StarIntel transport

The Docker worker consumes the current canonical target dispatch key:

  • documents.target.dispatch.<actor>

and retains the server's compatibility key:

  • actors.<actor>.new.target

Derived records publish only to documents.ingest.<dtype> on the durable documents topic exchange. There are no direct CouchDB writes.

WEF / YGL port

  • parses the current 2026 WEF semantic heading/category/list layout
  • retains legacy younggloballeaders.org card parsing
  • removes old guessed fname.lname@weforum.org emails
  • preserves raw source lines and source/evidence hashes
  • emits person, org, relation, and source v0.9 documents
  • fixture includes the comma-containing Hong Kong SAR, People's Republic of China case

Docker / Compose deployment

actors/compose.yaml is an independent worker plane that joins the existing starintel_backend network rather than modifying/restarting the core StarIntel stack.

  • custom Python 3.12 image with Playwright/Chromium 1.62.0 pinned together
  • unprivileged runtime user
  • Docker secret for the existing RabbitMQ password; no plaintext secret in .env or image
  • init: true and host IPC for Chromium
  • optional production Playwright seccomp override matched to the pinned version
  • actor-specific health check reaches RabbitMQ
  • immutable image tag (starintel/actor-wef-ygl:0.1.0 by default)
  • image-tag-only upgrade/rollback; no DB/server restart

See actors/DEPLOY.md.

Agent skills

  • actors/skills/create-website-scraper/SKILL.md
  • actors/skills/verify-website-scraper/SKILL.md
  • actors/skills/deploy-docker-actor/SKILL.md
  • adapter template included

The creation skill requires live-site research, fixture coverage, Docker/Compose registration, exact-image live verification, and prohibits CAPTCHA/fingerprint bypass and fabricated fields.

Verification already run locally

  • pytest -q: 6 passed
  • python -m compileall src tests: passed
  • YAML parse for actor Compose and workflow files: passed

Docker is not available in the local execution container, so the PR workflows are the authoritative image-build validation. GitHub CI builds the exact actor image and live-verifies WEF with Chromium; Forgejo has the image/fixture gate as well.

Scope

No edits to the core Lisp runtime, CouchDB/RabbitMQ services, or root docker-compose.yml. This is an independently deployable actor plane in the normal actor repository.

## Summary Moves the archived WEF / Young Global Leaders scraper port into the normal/public StarIntel actor home (`starintel-server`) and replaces the old Scrapy/shell deployment model with an extensible modern Python actor runtime plus an independent Docker Compose actor plane. The mistakenly-created `starintel-pro-actors` PR #14 is closed and superseded. ## Actor runtime - `actors/website/` is a standalone Python 3.12 actor package - asyncio mailbox actor model with bounded crawl scheduling - generic `SiteAdapter`, fetcher, registry, sink, request/batch/report contracts - third-party site plugins can register through `starintel.website_actors` entry points - HTTP-first `httpx`; Playwright Chromium fallback for JS shells, 401/403/429, and HTTP failures - one browser process per crawl with isolated browser contexts - images/media/fonts blocked for browser collection efficiency - deterministic StarIntel v0.9 IDs and provenance - JSONL standalone/debug sink and durable RabbitMQ ingest sink ## Current StarIntel transport The Docker worker consumes the current canonical target dispatch key: - `documents.target.dispatch.<actor>` and retains the server's compatibility key: - `actors.<actor>.new.target` Derived records publish only to `documents.ingest.<dtype>` on the durable `documents` topic exchange. There are no direct CouchDB writes. ## WEF / YGL port - parses the current 2026 WEF semantic heading/category/list layout - retains legacy `younggloballeaders.org` card parsing - removes old guessed `fname.lname@weforum.org` emails - preserves raw source lines and source/evidence hashes - emits `person`, `org`, `relation`, and `source` v0.9 documents - fixture includes the comma-containing `Hong Kong SAR, People's Republic of China` case ## Docker / Compose deployment `actors/compose.yaml` is an independent worker plane that joins the existing `starintel_backend` network rather than modifying/restarting the core StarIntel stack. - custom Python 3.12 image with Playwright/Chromium 1.62.0 pinned together - unprivileged runtime user - Docker secret for the existing RabbitMQ password; no plaintext secret in `.env` or image - `init: true` and host IPC for Chromium - optional production Playwright seccomp override matched to the pinned version - actor-specific health check reaches RabbitMQ - immutable image tag (`starintel/actor-wef-ygl:0.1.0` by default) - image-tag-only upgrade/rollback; no DB/server restart See `actors/DEPLOY.md`. ## Agent skills - `actors/skills/create-website-scraper/SKILL.md` - `actors/skills/verify-website-scraper/SKILL.md` - `actors/skills/deploy-docker-actor/SKILL.md` - adapter template included The creation skill requires live-site research, fixture coverage, Docker/Compose registration, exact-image live verification, and prohibits CAPTCHA/fingerprint bypass and fabricated fields. ## Verification already run locally - `pytest -q`: 6 passed - `python -m compileall src tests`: passed - YAML parse for actor Compose and workflow files: passed Docker is not available in the local execution container, so the PR workflows are the authoritative image-build validation. GitHub CI builds the exact actor image and live-verifies WEF with Chromium; Forgejo has the image/fixture gate as well. ## Scope No edits to the core Lisp runtime, CouchDB/RabbitMQ services, or root `docker-compose.yml`. This is an independently deployable actor plane in the normal actor repository.
ci: load actor pytest config in Forgejo
Some checks failed
Website actors / test-image (pull_request) Failing after 3s
Details
Docstring coverage / doc-coverage (pull_request) Failing after 57s
Details
d4e4dabebe
nsaspy commented 2026-09-08 01:58:34 +00:00
Author
Owner
Copy link

NOPE closing as you was a dumbfuck gpt sol high

NOPE closing as you was a dumbfuck gpt sol high
Some checks failed
Website actors / test-image (pull_request) Failing after 3s
Details
Docstring coverage / doc-coverage (pull_request) Failing after 57s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/modern-website-actors:feat/modern-website-actors
git switch feat/modern-website-actors

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch master
git merge --no-ff feat/modern-website-actors
git switch feat/modern-website-actors
git rebase master
git switch master
git merge --ff-only feat/modern-website-actors
git switch feat/modern-website-actors
git rebase master
git switch master
git merge --no-ff feat/modern-website-actors
git switch master
git merge --squash feat/modern-website-actors
git switch master
git merge --ff-only feat/modern-website-actors
git switch master
git merge feat/modern-website-actors
git push origin master
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug
Something isn't working

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
question
Further information is requested

  
wontfix
This will not be worked on

No labels
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
nsaspy/starintel-server!145
No description provided.