[P0-27] Introduce workload-specific dispatchers and cl-gserver router pools #35

New issue

Open

opened 2026-07-22 00:15:19 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-07-22 00:15:19 +00:00

(Migrated from github.com)

Problem

Most actors use the default shared dispatcher, blocking database work is mixed with control/routing work, producer/CouchDB agents use pinned threads, and Rabbit consumers bypass the actor routing model through ad hoc thread creation.

The canonical auto-research architecture requires router-backed routee pools for hot in-process paths and separate control/data dispatchers.

Required dispatcher tree

At minimum:

:platform/control — lifecycle, registry, health, routee management, overload decisions
:platform/ingest — validation and ingest coordination
:platform/storage — blocking CouchDB operations
:platform/rabbit — Rabbit owner/reactor control
:platform/target — target scheduling and lease coordination
:platform/cpu — bounded CPU-heavy transforms
:platform/research — resumable research job coordination

Required router pools

Create managed or explicitly lifecycle-owned cl-gserver routers for:

document validation/codec
persistence operations
target dispatch
search/query work
normalization/extraction
research jobs

Use round-robin for stateless equivalent routees and keyed routing where target/dataset ordering or state affinity matters. Do not use random routing for stateful keyed work.

Acceptance tests

Saturating storage work does not block control/health/lease renewal.
Routee pools preserve one-message-at-a-time actor state access.
Pool sizes consume the global thread budget rather than creating threads directly.
Router membership, health, and queue pressure are observable.
Benchmarks compare default shared, custom shared, pinned, and router configurations before adopting defaults.

## Problem Most actors use the default shared dispatcher, blocking database work is mixed with control/routing work, producer/CouchDB agents use pinned threads, and Rabbit consumers bypass the actor routing model through ad hoc thread creation. The canonical auto-research architecture requires router-backed routee pools for hot in-process paths and separate control/data dispatchers. ## Required dispatcher tree At minimum: - `:platform/control` — lifecycle, registry, health, routee management, overload decisions - `:platform/ingest` — validation and ingest coordination - `:platform/storage` — blocking CouchDB operations - `:platform/rabbit` — Rabbit owner/reactor control - `:platform/target` — target scheduling and lease coordination - `:platform/cpu` — bounded CPU-heavy transforms - `:platform/research` — resumable research job coordination ## Required router pools Create managed or explicitly lifecycle-owned cl-gserver routers for: - document validation/codec - persistence operations - target dispatch - search/query work - normalization/extraction - research jobs Use round-robin for stateless equivalent routees and keyed routing where target/dataset ordering or state affinity matters. Do not use random routing for stateful keyed work. ## Acceptance tests - Saturating storage work does not block control/health/lease renewal. - Routee pools preserve one-message-at-a-time actor state access. - Pool sizes consume the global thread budget rather than creating threads directly. - Router membership, health, and queue pressure are observable. - Benchmarks compare default shared, custom shared, pinned, and router configurations before adopting defaults.