nsaspy/starintel-auto-research

Fork 0

DESIGN_READY_FOR_OPERATOR_REVIEW: StarIntel runtime thread-budget authority #163

New issue

Open

opened 2026-08-29 10:28:05 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-08-29 10:28:05 +00:00

Owner

Authority / state

Source research: #176
Owning implementation issue: lost-rob0t/starintel-server#34
Related P0 runtime work: starintel-server#35, #36, #37
Current server master inspected: e99aadd3f44505bc692708aad5e34ab0c4ab0036
Research state: READY_FOR_DESIGN
Design state: DESIGN_READY_FOR_OPERATOR_REVIEW
Implementation approval: PENDING / AWAITING_OPERATOR_APPROVAL — operator only

This design is not executable by RAGE until the operator explicitly approves implementation.

Design goal

Create one runtime-owned thread-budget authority that validates and allocates every StarIntel-owned OS-thread demand before long-lived components start.

The fix is not a global executor. Workload-specific isolation remains, but every pool consumes capacity from one process-level budget.

Ownership

`starintel-server` runtime/component layer owns

budget configuration and validation;
component thread-demand registration;
deterministic allocation plan;
creation of runtime-owned executors/dispatchers/worker groups from granted allocations;
live owned-thread accounting;
shutdown/lifecycle mapping for each allocation.

Individual components own

declaring min/desired/max demand and workload class;
using only granted allocation;
reporting live owned-thread identities through runtime instrumentation.

StarLang owns no new syntax

StarLang actors/domain servers may carry resource-profile metadata where useful, but host runtime remains authoritative. A model/source definition cannot increase process thread authority.

Process-wide authority

Use canonical Common Lisp special variables as the process-wide thread-limit authority:

(defparameter *max-process-threads* 32)
(defparameter *reserved-control-threads* 4)

These values are process invariants, not a nested runtime configuration object. Runtime components read the canonical specials through the budget planner and may not independently reinterpret them as per-component capacity.

If the server init/config system later exposes overrides, the loader initializes/sets these specials. The parsed config object is not itself the runtime authority and no component receives an independent thread-budget config object.

Component-specific desired counts may remain explicit, but are interpreted only as demand inputs to the process-wide planner.

Closed demand classes

At minimum:

control
actor-data
rabbit
storage
cpu
http
scheduler
transport
pinned
fixed-overhead

Unknown classes fail validation rather than becoming untracked pools.

Resource-plan API

Conceptual Common Lisp API:

register-thread-demand(runtime, component-id, class,
                       min, desired, max, fixed?)

compute-thread-plan(max-process-threads, reserved-control-threads, demands)
  -> thread-plan | unsatisfiable-budget

grant-for(plan, component-id)
  -> positive integer allocation

register-owned-thread(runtime, component-id, thread-ref)
unregister-owned-thread(runtime, component-id, thread-ref)

The planner is pure/deterministic and testable without starting threads. Production callers normally pass the canonical values from *max-process-threads* and *reserved-control-threads*.

Startup sequence

initialize process-wide runtime specials;
build runtime/component graph;
collect every component thread demand without starting long-lived resources;
compute one thread plan from the canonical process-wide limits;
reject unsatisfiable plan before network listeners/connections/actors become active;
realize components from explicit grants;
compare observed owned-thread accounting against the plan during readiness;
expose plan + live usage through diagnostics/health.

Partial startup rollback remains #37 lifecycle behavior, but thread-plan validation should happen before rollback is needed wherever possible.

Deterministic allocation v1

Keep allocation simple:

reserve declared fixed overhead;
reserve control minimum;
satisfy component minima;
fail if total minima exceed ceiling;
allocate remaining desired capacity in deterministic configured order or explicit weight policy;
cap at each component maximum;
no runtime autoscaling beyond the plan in v1.

Any rounding/remainder rule must be deterministic and covered by fixtures.

Dispatcher integration

The reserved :pinned Sento semantic collision must be removed. Custom workload dispatchers introduced by #35 receive worker counts from the plan.

A component may choose a Sento/cl-gserver/lparallel/Hunchentoot executor implementation appropriate to its workload, but executor construction must consume its grant and register resulting owned threads.

No subsystem may treat the same global knob as independent capacity.

Rabbit integration

Rabbit owner/reactor threads and processing workers are separate budget demands.

Consumer prefetch should later be derived by #36 from granted processing capacity plus bounded per-worker in-flight policy rather than remaining an unrelated hard-coded number.

HTTP integration

Hunchentoot worker/thread limits consume the HTTP allocation. HTTP configuration cannot independently raise process capacity above the runtime ceiling.

If the HTTP implementation internally creates unavoidable fixed housekeeping threads, declare/document them as fixed overhead and verify them in the observed-count fixture.

Timer/scheduler integration

Prefer shared timer/scheduler resources when current libraries permit it, but do not redesign scheduler semantics in this slice. Any dedicated runtime-owned timer threads must be registered and counted.

Control-reserve invariant

Data-plane allocations may not consume *reserved-control-threads*.

The reserve exists for:

supervision/lifecycle;
health/readiness;
shutdown/drain coordination;
lease-renewal/control paths where relevant;
overload decisions.

This design does not promise every control action gets a dedicated OS thread; it guarantees data-plane planning cannot consume all execution capacity.

Live accounting

Expose at least:

max threads
planned runtime-owned threads
live runtime-owned threads
unattributed runtime-owned threads (must normally be 0)
allocation by component/class

The invariant is based on runtime-owned threads. SBCL GC/runtime threads or unavoidable foreign-library threads must be separately documented rather than silently counted/un-counted depending on timing.

Adversarial review

Rejected:

one giant shared executor for control + blocking storage + HTTP;
post-start warning instead of pre-start rejection;
per-component self-sizing without central grants;
dynamic autoscaling before fixed planning is correct;
counting only named pools while ignoring owner/timer/reactor/pinned threads;
permitting raw bt:make-thread in arbitrary product components;
treating actor count as thread count (shared dispatchers and pinned actors differ);
allocating all capacity to data-plane desired counts and hoping health/shutdown remain responsive;
making a nested (:runtime ...) configuration object the canonical process authority when two process-wide Common Lisp specials are sufficient.

Dependency-ordered implementation slices

All remain AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL.

Pure thread-demand + plan model/validator — no production thread wiring yet.
Process-wide specials + fail-fast startup gate — collect current known demands and reject over-budget runtime before side effects.
Executor/dispatcher grant wiring — Sento/lparallel/Rabbit/HTTP current pools consume plan allocations; remove custom :pinned collision.
Owned-thread registry + observed accounting — diagnostics and leak/unattributed checks.
#35 integration — workload dispatchers registered through budget authority.
#36/#37 integration — admission/prefetch capacity and lifecycle cleanup consume/report the same plan.

Mandatory RED-first targets

Slice 1

Before any production mutation, add pure deterministic fixtures:

max=8, fixed/control minima + component minima total 10 -> expected unsatisfiable-budget; untouched code fails because no planner exists;
same global ingest-workers=4 consumed by three independent current subsystems -> fixture demonstrates intended total budget violation instead of treating each 4 as independently valid;
deterministic exact-fit configuration -> expected grants sum exactly to ceiling;
unknown/unregistered thread-producing component/class -> rejected.

Slice 2

Startup fixture with *max-process-threads* intentionally below the component graph's required minima must fail before any network/listener/thread side effects. Untouched current startup should fail the test because it has no central preflight gate.

Slice 4

A fake runtime-owned-thread adapter must report an extra unregistered thread and cause readiness/accounting verification to fail.

Acceptance

one canonical *max-process-threads* process ceiling controls all StarIntel-owned execution sources;
*reserved-control-threads* protects control capacity from data-plane allocation;
unsatisfiable runtime demand fails before long-lived startup;
ingest-workers or any successor knob cannot multiply independently across subsystems;
every runtime-created thread has an owning component/allocation;
observed owned-thread count matches plan within explicitly declared fixed overhead;
#35 dispatchers consume budget rather than bypassing it;
#36 can derive admission/prefetch limits from granted processing capacity;
#37 can stop/join exactly runtime-owned resources;
no new StarLang syntax, nested thread-budget configuration authority, or external Python controller is introduced.

Implementation approval

PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL

Only the operator may authorize implementation.

## Authority / state - Source research: #176 - Owning implementation issue: `lost-rob0t/starintel-server#34` - Related P0 runtime work: `starintel-server#35`, `#36`, `#37` - Current server master inspected: `e99aadd3f44505bc692708aad5e34ab0c4ab0036` - Research state: `READY_FOR_DESIGN` - Design state: `DESIGN_READY_FOR_OPERATOR_REVIEW` - **Implementation approval: PENDING / AWAITING_OPERATOR_APPROVAL — operator only** This design is not executable by RAGE until the operator explicitly approves implementation. ## Design goal Create one runtime-owned thread-budget authority that validates and allocates every StarIntel-owned OS-thread demand before long-lived components start. The fix is not a global executor. Workload-specific isolation remains, but every pool consumes capacity from one process-level budget. ## Ownership ### `starintel-server` runtime/component layer owns - budget configuration and validation; - component thread-demand registration; - deterministic allocation plan; - creation of runtime-owned executors/dispatchers/worker groups from granted allocations; - live owned-thread accounting; - shutdown/lifecycle mapping for each allocation. ### Individual components own - declaring min/desired/max demand and workload class; - using only granted allocation; - reporting live owned-thread identities through runtime instrumentation. ### StarLang owns no new syntax StarLang actors/domain servers may carry resource-profile metadata where useful, but host runtime remains authoritative. A model/source definition cannot increase process thread authority. ## Process-wide authority Use canonical Common Lisp special variables as the process-wide thread-limit authority: ```lisp (defparameter *max-process-threads* 32) (defparameter *reserved-control-threads* 4) ``` These values are process invariants, not a nested runtime configuration object. Runtime components read the canonical specials through the budget planner and may not independently reinterpret them as per-component capacity. If the server init/config system later exposes overrides, the loader initializes/sets these specials. The parsed config object is not itself the runtime authority and no component receives an independent thread-budget config object. Component-specific desired counts may remain explicit, but are interpreted only as demand inputs to the process-wide planner. ## Closed demand classes At minimum: - `control` - `actor-data` - `rabbit` - `storage` - `cpu` - `http` - `scheduler` - `transport` - `pinned` - `fixed-overhead` Unknown classes fail validation rather than becoming untracked pools. ## Resource-plan API Conceptual Common Lisp API: ```text register-thread-demand(runtime, component-id, class, min, desired, max, fixed?) compute-thread-plan(max-process-threads, reserved-control-threads, demands) -> thread-plan | unsatisfiable-budget grant-for(plan, component-id) -> positive integer allocation register-owned-thread(runtime, component-id, thread-ref) unregister-owned-thread(runtime, component-id, thread-ref) ``` The planner is pure/deterministic and testable without starting threads. Production callers normally pass the canonical values from `*max-process-threads*` and `*reserved-control-threads*`. ## Startup sequence 1. initialize process-wide runtime specials; 2. build runtime/component graph; 3. collect every component thread demand without starting long-lived resources; 4. compute one thread plan from the canonical process-wide limits; 5. reject unsatisfiable plan before network listeners/connections/actors become active; 6. realize components from explicit grants; 7. compare observed owned-thread accounting against the plan during readiness; 8. expose plan + live usage through diagnostics/health. Partial startup rollback remains #37 lifecycle behavior, but thread-plan validation should happen before rollback is needed wherever possible. ## Deterministic allocation v1 Keep allocation simple: - reserve declared fixed overhead; - reserve control minimum; - satisfy component minima; - fail if total minima exceed ceiling; - allocate remaining desired capacity in deterministic configured order or explicit weight policy; - cap at each component maximum; - no runtime autoscaling beyond the plan in v1. Any rounding/remainder rule must be deterministic and covered by fixtures. ## Dispatcher integration The reserved `:pinned` Sento semantic collision must be removed. Custom workload dispatchers introduced by #35 receive worker counts from the plan. A component may choose a Sento/cl-gserver/lparallel/Hunchentoot executor implementation appropriate to its workload, but executor construction must consume its grant and register resulting owned threads. No subsystem may treat the same global knob as independent capacity. ## Rabbit integration Rabbit owner/reactor threads and processing workers are separate budget demands. Consumer prefetch should later be derived by #36 from granted processing capacity plus bounded per-worker in-flight policy rather than remaining an unrelated hard-coded number. ## HTTP integration Hunchentoot worker/thread limits consume the HTTP allocation. HTTP configuration cannot independently raise process capacity above the runtime ceiling. If the HTTP implementation internally creates unavoidable fixed housekeeping threads, declare/document them as fixed overhead and verify them in the observed-count fixture. ## Timer/scheduler integration Prefer shared timer/scheduler resources when current libraries permit it, but do not redesign scheduler semantics in this slice. Any dedicated runtime-owned timer threads must be registered and counted. ## Control-reserve invariant Data-plane allocations may not consume `*reserved-control-threads*`. The reserve exists for: - supervision/lifecycle; - health/readiness; - shutdown/drain coordination; - lease-renewal/control paths where relevant; - overload decisions. This design does not promise every control action gets a dedicated OS thread; it guarantees data-plane planning cannot consume all execution capacity. ## Live accounting Expose at least: ```text max threads planned runtime-owned threads live runtime-owned threads unattributed runtime-owned threads (must normally be 0) allocation by component/class ``` The invariant is based on **runtime-owned** threads. SBCL GC/runtime threads or unavoidable foreign-library threads must be separately documented rather than silently counted/un-counted depending on timing. ## Adversarial review Rejected: 1. one giant shared executor for control + blocking storage + HTTP; 2. post-start warning instead of pre-start rejection; 3. per-component self-sizing without central grants; 4. dynamic autoscaling before fixed planning is correct; 5. counting only named pools while ignoring owner/timer/reactor/pinned threads; 6. permitting raw `bt:make-thread` in arbitrary product components; 7. treating actor count as thread count (shared dispatchers and pinned actors differ); 8. allocating all capacity to data-plane desired counts and hoping health/shutdown remain responsive; 9. making a nested `(:runtime ...)` configuration object the canonical process authority when two process-wide Common Lisp specials are sufficient. ## Dependency-ordered implementation slices All remain **AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL**. 1. **Pure thread-demand + plan model/validator** — no production thread wiring yet. 2. **Process-wide specials + fail-fast startup gate** — collect current known demands and reject over-budget runtime before side effects. 3. **Executor/dispatcher grant wiring** — Sento/lparallel/Rabbit/HTTP current pools consume plan allocations; remove custom `:pinned` collision. 4. **Owned-thread registry + observed accounting** — diagnostics and leak/unattributed checks. 5. **#35 integration** — workload dispatchers registered through budget authority. 6. **#36/#37 integration** — admission/prefetch capacity and lifecycle cleanup consume/report the same plan. ## Mandatory RED-first targets ### Slice 1 Before any production mutation, add pure deterministic fixtures: - `max=8`, fixed/control minima + component minima total 10 -> expected `unsatisfiable-budget`; untouched code fails because no planner exists; - same global `ingest-workers=4` consumed by three independent current subsystems -> fixture demonstrates intended total budget violation instead of treating each 4 as independently valid; - deterministic exact-fit configuration -> expected grants sum exactly to ceiling; - unknown/unregistered thread-producing component/class -> rejected. ### Slice 2 Startup fixture with `*max-process-threads*` intentionally below the component graph's required minima must fail before any network/listener/thread side effects. Untouched current startup should fail the test because it has no central preflight gate. ### Slice 4 A fake runtime-owned-thread adapter must report an extra unregistered thread and cause readiness/accounting verification to fail. ## Acceptance - one canonical `*max-process-threads*` process ceiling controls all StarIntel-owned execution sources; - `*reserved-control-threads*` protects control capacity from data-plane allocation; - unsatisfiable runtime demand fails before long-lived startup; - `ingest-workers` or any successor knob cannot multiply independently across subsystems; - every runtime-created thread has an owning component/allocation; - observed owned-thread count matches plan within explicitly declared fixed overhead; - #35 dispatchers consume budget rather than bypassing it; - #36 can derive admission/prefetch limits from granted processing capacity; - #37 can stop/join exactly runtime-owned resources; - no new StarLang syntax, nested thread-budget configuration authority, or external Python controller is introduced. ## Implementation approval `PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL` Only the operator may authorize implementation.

nsaspy referenced this issue

2026-08-29 10:28:07 +00:00

StarLang research: OSINT domain-server orchestration gaps #175