OSINT cluster research: SpiderFoot automation, event propagation, and correlation #177

New issue

Open

opened 2026-08-29 10:28:07 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-08-29 10:28:07 +00:00

Owner

Goal

Deeply research SpiderFoot as an OSINT automation architecture and extract the parts StarIntel needs without cloning its module catalog.

Architecture rule

Treat SpiderFoot primarily as research for an Investigation/Discovery domain-server cluster and shared orchestration semantics. Individual sources should still live in their proper domain servers.

Feature surface to verify

200+ modular collectors/analyzers
publisher/subscriber event propagation between modules
typed event consumption/emission
scan target types and scan presets
recursive discovery triggered by emitted events
YAML correlation engine and rule semantics
correlation evidence/explanations
module configuration and credential handling
provider/API integrations
internal extractors and analyzers
external-tool invocation
web/CLI/API execution surfaces
scan persistence and queryable result storage
custom querying
graph/visualization model
CSV/JSON/GEXF export
Tor/dark-web routing support
source module attribution
error/timeout/rate-limit behavior
monitoring/change-notification concepts from HX that are architecturally useful even if not copied

Key StarIntel questions

Research whether StarLang needs or already has:

typed event subscriptions
data-driven capability activation from newly emitted documents
bounded recursive expansion
declarative correlation rules
explainable rule firing
scan/job state and resumability
dynamic capability filtering based on target/document type
safe external-tool adapter invocation

Output target

Map reusable behavior into domain-server orchestration and shared StarLang runtime semantics. Do not create spiderfoot-* actors for each module.

Implementation preference

StarLang → extend StarLang → Common Lisp → Python only as last resort.

Output gate

Stage findings in roam/internal/OSINT-TOOLS-LEDGER.org and this issue until explicit instruction to add research to ARARD/ARADR files.

## Goal Deeply research SpiderFoot as an OSINT automation architecture and extract the parts StarIntel needs without cloning its module catalog. ## Architecture rule Treat SpiderFoot primarily as research for an Investigation/Discovery domain-server cluster and shared orchestration semantics. Individual sources should still live in their proper domain servers. ## Feature surface to verify - 200+ modular collectors/analyzers - publisher/subscriber event propagation between modules - typed event consumption/emission - scan target types and scan presets - recursive discovery triggered by emitted events - YAML correlation engine and rule semantics - correlation evidence/explanations - module configuration and credential handling - provider/API integrations - internal extractors and analyzers - external-tool invocation - web/CLI/API execution surfaces - scan persistence and queryable result storage - custom querying - graph/visualization model - CSV/JSON/GEXF export - Tor/dark-web routing support - source module attribution - error/timeout/rate-limit behavior - monitoring/change-notification concepts from HX that are architecturally useful even if not copied ## Key StarIntel questions Research whether StarLang needs or already has: - typed event subscriptions - data-driven capability activation from newly emitted documents - bounded recursive expansion - declarative correlation rules - explainable rule firing - scan/job state and resumability - dynamic capability filtering based on target/document type - safe external-tool adapter invocation ## Output target Map reusable behavior into domain-server orchestration and shared StarLang runtime semantics. Do not create `spiderfoot-*` actors for each module. ## Implementation preference StarLang → extend StarLang → Common Lisp → Python only as last resort. ## Output gate Stage findings in `roam/internal/OSINT-TOOLS-LEDGER.org` and this issue until explicit instruction to add research to ARARD/ARADR files.

nsaspy referenced this issue

2026-09-02 17:26:16 +00:00

DESIGN_READY_FOR_OPERATOR_REVIEW: managed runtime components, shutdown deadline, and reconnect fencing #200

nsaspy referenced this issue

2026-09-02 17:26:16 +00:00

P0 ARDR research: finish managed runtime lifecycle, deadlines, and reconnect supervision #201