[P0-26] Enforce one total process thread budget across actors, consumers, tasks, HTTP, timers, and transports #34

New issue

Open

opened 2026-07-22 00:15:03 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-07-22 00:15:03 +00:00

(Migrated from github.com)

Governance / ARDR state

Research authority: lost-rob0t/starintel-auto-research#176 — READY_FOR_DESIGN
Design authority: lost-rob0t/starintel-auto-research#177 — DESIGN_READY_FOR_OPERATOR_REVIEW
Related runtime dependencies: #35 workload dispatchers, #36 bounded admission/mailboxes/prefetch, #37 lifecycle/thread ownership
Implementation approval: PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL

This issue must not enter the executable RAGE queue until the operator explicitly approves implementation.

Confirmed defect

*ingest-workers* is independently reused for several subsystems instead of enforcing a total limit:

default Sento shared dispatcher workers
an additional custom dispatcher named :pinned
three Rabbit consumer groups, each creating *ingest-workers* OS threads
event consumers
a separate lparallel kernel
producer and CouchDB pinned agents
target/GC/timeout/scheduler timers
Hunchentoot configured for up to 50 threads

This allows configuration to multiply threads rather than cap them.

The custom dispatcher named :pinned is especially broken: Sento treats dispatcher id :pinned as the built-in one-thread-per-actor mailbox before looking up custom dispatchers. The configured custom :pinned worker pool is therefore unused overhead, while comments incorrectly claim agent thread affinity.

Design decision

Introduce one runtime-owned thread-budget authority. Every thread-producing StarIntel component registers min/desired/max demand before long-lived startup. A pure deterministic planner computes grants under the canonical process-wide Common Lisp limits, reserves control capacity, and rejects unsatisfiable configurations before network/listener/actor side effects.

The process-wide authority is:

(defparameter *max-process-threads* 32)
(defparameter *reserved-control-threads* 4)

These specials are the runtime authority. Do not introduce a nested (:runtime ...) thread-budget object. If init/config later exposes overrides, it initializes/sets these symbols rather than becoming a competing source of truth.

Workload-specific dispatchers remain separate; they consume grants from this authority rather than becoming a single giant executor.

No new StarLang syntax is required. StarLang-defined actors/domain servers cannot create or expand host thread authority. Common Lisp runtime/component code owns planning, executor construction, accounting and enforcement.

Proposed implementation slices

All remain AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL.

Pure thread-demand model + deterministic budget planner/validator.
Canonical process-wide specials + pre-start fail-fast budget gate.
Wire Sento/lparallel/Rabbit/Hunchentoot/current pools to explicit grants; remove the custom :pinned collision.
Runtime-owned thread registry + planned-vs-live accounting/readiness check.
#35 workload-dispatcher integration through the same budget authority.
#36/#37 admission/prefetch and lifecycle cleanup integration.

Mandatory RED-first gate

No production mutation before a valid failing fixture exists.

First RED

Add a pure deterministic fixture where:

*max-process-threads* = 8;
fixed/control minima + component minima require 10 threads;
expected result is a typed unsatisfiable-budget before any thread/network side effect.

Untouched current code must fail because no central planner exists.

Also add a regression fixture proving one ingest-workers = 4 value consumed independently by three current subsystems produces a total demand greater than the intended ceiling rather than being treated as three unrelated valid 4-worker settings.

Negative fixtures must reject:

unknown/unregistered thread demand classes;
negative/zero invalid allocations where a worker is required;
component minimum greater than maximum;
a runtime-owned thread appearing without an owning allocation.

Required changes

Add canonical *max-process-threads* and *reserved-control-threads* specials plus a pure startup thread-budget calculator.
Count shared dispatcher workers, pinned actors, Rabbit owner/reactor threads, task workers, HTTP workers, timers, KV, and transport threads.
Fail startup when required runtime demand exceeds *max-process-threads*; optionally warn only in explicit development mode where the design allows it.
Remove the unused custom :pinned dispatcher.
Derive subsystem allocations from one budget, with *reserved-control-threads* unavailable to data-plane desired allocations.
Expose configured/planned and live runtime-owned thread counts.
Require runtime components to obtain explicit grants before constructing thread pools/workers.
Do not add a nested thread-budget configuration authority; config loaders may only initialize the canonical specials.

Acceptance tests

Startup fails for an over-budget configuration before long-lived side effects.
Increasing ingest concurrency cannot multiply every subsystem independently.
Exact-fit allocations are deterministic and sum to the configured ceiling.
Reserved control capacity cannot be consumed by data-plane desired allocations.
Reported budget equals observed runtime-owned threads within documented fixed overhead.
No runtime component creates an unregistered thread.
#35 dispatchers consume allocations from this budget rather than independently sizing pools.
#36 can derive admission/prefetch limits from granted processing capacity.
#37 can stop/join exactly the runtime-owned resources represented by the plan.

Implementation approval

PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL

ARDR/ADARD cannot change this state. Explicit operator approval is required before RAGE implementation.

## Governance / ARDR state - Research authority: `lost-rob0t/starintel-auto-research#176` — `READY_FOR_DESIGN` - Design authority: `lost-rob0t/starintel-auto-research#177` — `DESIGN_READY_FOR_OPERATOR_REVIEW` - Related runtime dependencies: #35 workload dispatchers, #36 bounded admission/mailboxes/prefetch, #37 lifecycle/thread ownership - **Implementation approval: PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL** This issue must not enter the executable RAGE queue until the operator explicitly approves implementation. ## Confirmed defect `*ingest-workers*` is independently reused for several subsystems instead of enforcing a total limit: - default Sento shared dispatcher workers - an additional custom dispatcher named `:pinned` - three Rabbit consumer groups, each creating `*ingest-workers*` OS threads - event consumers - a separate lparallel kernel - producer and CouchDB pinned agents - target/GC/timeout/scheduler timers - Hunchentoot configured for up to 50 threads This allows configuration to multiply threads rather than cap them. The custom dispatcher named `:pinned` is especially broken: Sento treats dispatcher id `:pinned` as the built-in one-thread-per-actor mailbox before looking up custom dispatchers. The configured custom `:pinned` worker pool is therefore unused overhead, while comments incorrectly claim agent thread affinity. ## Design decision Introduce one runtime-owned thread-budget authority. Every thread-producing StarIntel component registers min/desired/max demand before long-lived startup. A pure deterministic planner computes grants under the canonical process-wide Common Lisp limits, reserves control capacity, and rejects unsatisfiable configurations before network/listener/actor side effects. The process-wide authority is: ```lisp (defparameter *max-process-threads* 32) (defparameter *reserved-control-threads* 4) ``` These specials are the runtime authority. Do not introduce a nested `(:runtime ...)` thread-budget object. If init/config later exposes overrides, it initializes/sets these symbols rather than becoming a competing source of truth. Workload-specific dispatchers remain separate; they consume grants from this authority rather than becoming a single giant executor. No new StarLang syntax is required. StarLang-defined actors/domain servers cannot create or expand host thread authority. Common Lisp runtime/component code owns planning, executor construction, accounting and enforcement. ## Proposed implementation slices All remain **AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL**. 1. Pure thread-demand model + deterministic budget planner/validator. 2. Canonical process-wide specials + pre-start fail-fast budget gate. 3. Wire Sento/lparallel/Rabbit/Hunchentoot/current pools to explicit grants; remove the custom `:pinned` collision. 4. Runtime-owned thread registry + planned-vs-live accounting/readiness check. 5. #35 workload-dispatcher integration through the same budget authority. 6. #36/#37 admission/prefetch and lifecycle cleanup integration. ## Mandatory RED-first gate No production mutation before a valid failing fixture exists. ### First RED Add a pure deterministic fixture where: - `*max-process-threads* = 8`; - fixed/control minima + component minima require 10 threads; - expected result is a typed `unsatisfiable-budget` before any thread/network side effect. Untouched current code must fail because no central planner exists. Also add a regression fixture proving one `ingest-workers = 4` value consumed independently by three current subsystems produces a total demand greater than the intended ceiling rather than being treated as three unrelated valid `4`-worker settings. Negative fixtures must reject: - unknown/unregistered thread demand classes; - negative/zero invalid allocations where a worker is required; - component minimum greater than maximum; - a runtime-owned thread appearing without an owning allocation. ## Required changes - Add canonical `*max-process-threads*` and `*reserved-control-threads*` specials plus a pure startup thread-budget calculator. - Count shared dispatcher workers, pinned actors, Rabbit owner/reactor threads, task workers, HTTP workers, timers, KV, and transport threads. - Fail startup when required runtime demand exceeds `*max-process-threads*`; optionally warn only in explicit development mode where the design allows it. - Remove the unused custom `:pinned` dispatcher. - Derive subsystem allocations from one budget, with `*reserved-control-threads*` unavailable to data-plane desired allocations. - Expose configured/planned and live runtime-owned thread counts. - Require runtime components to obtain explicit grants before constructing thread pools/workers. - Do not add a nested thread-budget configuration authority; config loaders may only initialize the canonical specials. ## Acceptance tests - Startup fails for an over-budget configuration before long-lived side effects. - Increasing ingest concurrency cannot multiply every subsystem independently. - Exact-fit allocations are deterministic and sum to the configured ceiling. - Reserved control capacity cannot be consumed by data-plane desired allocations. - Reported budget equals observed runtime-owned threads within documented fixed overhead. - No runtime component creates an unregistered thread. - #35 dispatchers consume allocations from this budget rather than independently sizing pools. - #36 can derive admission/prefetch limits from granted processing capacity. - #37 can stop/join exactly the runtime-owned resources represented by the plan. ## Implementation approval `PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL` ARDR/ADARD cannot change this state. Explicit operator approval is required before RAGE implementation.