Research collection actors and design shared source runtime #14

Merged

lost-rob0t merged 3 commits from research/collection-actors-002 into agent/adaptive-research-dag

2026-07-20 00:20:23 +00:00

lost-rob0t commented

2026-07-20 00:11:29 +00:00

(Migrated from github.com)

Loop 002: Collection Actors

This stacked draft PR advances the canonical fourth-generation research loop from OSINT platform comparison into the complete collection-actor inventory.

Base dependency: agent/adaptive-research-dag / PR #13.

Research

Add STAR-RESEARCH-002 Collection Actor Architecture with current official-source analysis of:

OpenFEC API, bulk files, filings, and amendment-oriented synchronization
Telegram Bot API versus full client / TDLib collection
Mastodon per-instance REST and streaming behavior
Bluesky / AT Protocol repository synchronization, signed commits, firehose cursors, and resynchronization
Discord Gateway sequence state, intents, and dynamic rate limits
Reddit access and source-policy lifecycle
LinkedIn product- and partner-scoped access
Scrapy engine, middleware, spiders, and item pipelines
Playwright browser-context isolation, tracing, network capture, and download persistence
OpenCTI connector lifecycle and Singer incremental state patterns

The canonical source list is classified into eight execution classes:

snapshot import
paged pull
incremental pull
event stream
repository synchronization
interactive session
enrichment
bounded tool execution

Proposed design

Add draft STAR-COLLECT-001 Collection Actor Contract and Source Runtime.

It defines:

collector definitions projected into the capability catalog
configured source instances
immutable source-policy snapshots
collection requests and durable run state machines
raw observations before normalization
source-specific checkpoint payloads inside a common envelope
atomic collection-batch commits
rate and monetary budgets
scope enforcement
credential isolation and revocation
in-process Sento actors and external Star Router / ZeroMQ workers
shared mock and replay conformance tests

The design intentionally does not implement every platform collector at once. Its first adapter phase uses a small set covering distinct execution classes.

Research-loop state

close Loop 001 after the user's explicit instruction to continue
mark Loop 002 as review
queue Loop 003: document specification extensions
leave both proposed runtime designs unimplemented and independently subject to approval
do not alter roam/implement/ or status ledgers

Validation

branch is 3 commits ahead of agent/adaptive-research-dag and 0 behind
two Org-roam nodes added and the canonical project index updated
the original fourth-generation idea backlog remains unchanged
only collection-runtime research and design files are added
local scripts/sync.py --check, Pages export, and link audit cannot be run through the connector; PR workflows must validate the generated site and EPUB

## Loop 002: Collection Actors This stacked draft PR advances the canonical fourth-generation research loop from OSINT platform comparison into the complete collection-actor inventory. Base dependency: `agent/adaptive-research-dag` / PR #13. ## Research Add `STAR-RESEARCH-002 Collection Actor Architecture` with current official-source analysis of: - OpenFEC API, bulk files, filings, and amendment-oriented synchronization - Telegram Bot API versus full client / TDLib collection - Mastodon per-instance REST and streaming behavior - Bluesky / AT Protocol repository synchronization, signed commits, firehose cursors, and resynchronization - Discord Gateway sequence state, intents, and dynamic rate limits - Reddit access and source-policy lifecycle - LinkedIn product- and partner-scoped access - Scrapy engine, middleware, spiders, and item pipelines - Playwright browser-context isolation, tracing, network capture, and download persistence - OpenCTI connector lifecycle and Singer incremental state patterns The canonical source list is classified into eight execution classes: - snapshot import - paged pull - incremental pull - event stream - repository synchronization - interactive session - enrichment - bounded tool execution ## Proposed design Add draft `STAR-COLLECT-001 Collection Actor Contract and Source Runtime`. It defines: - collector definitions projected into the capability catalog - configured source instances - immutable source-policy snapshots - collection requests and durable run state machines - raw observations before normalization - source-specific checkpoint payloads inside a common envelope - atomic collection-batch commits - rate and monetary budgets - scope enforcement - credential isolation and revocation - in-process Sento actors and external Star Router / ZeroMQ workers - shared mock and replay conformance tests The design intentionally does not implement every platform collector at once. Its first adapter phase uses a small set covering distinct execution classes. ## Research-loop state - close Loop 001 after the user's explicit instruction to continue - mark Loop 002 as review - queue Loop 003: document specification extensions - leave both proposed runtime designs unimplemented and independently subject to approval - do not alter `roam/implement/` or status ledgers ## Validation - branch is 3 commits ahead of `agent/adaptive-research-dag` and 0 behind - two Org-roam nodes added and the canonical project index updated - the original fourth-generation idea backlog remains unchanged - only collection-runtime research and design files are added - local `scripts/sync.py --check`, Pages export, and link audit cannot be run through the connector; PR workflows must validate the generated site and EPUB