nsaspy/starintel-auto-research

Fork 0

DESIGN_READY_FOR_OPERATOR_REVIEW: managed runtime components, shutdown deadline, and reconnect fencing #200

New issue

Open

opened 2026-09-02 17:26:16 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-02 17:26:16 +00:00

Owner

Authority / state

Source research: #182
Owning implementation issue: lost-rob0t/starintel-server#37
Related P0 design: #177 thread-budget authority
Related P0 issues: starintel-server#35, #36
Current server master inspected: 8fb297d146e7332fae7e38170b5b49d49530ac53
Research state: READY_FOR_DESIGN
Design state: DESIGN_READY_FOR_OPERATOR_REVIEW
Implementation approval: PENDING / AWAITING_OPERATOR_APPROVAL — operator only

This design must not enter executable RAGE until explicit operator implementation approval.

Design goal

Extend the existing star.runtime lifecycle owner into a closed component/resource supervisor. Do not replace it and do not introduce a second service manager.

The resulting runtime must:

know every owned component/resource/thread;
start in dependency order and roll back deterministically;
quiesce/drain/stop under one absolute process shutdown deadline;
report incomplete cleanup rather than silently calling it clean;
own bounded reconnect for reconnectable dependencies;
expose readiness from component health rather than retained handle presence;
provide the owner identity consumed by #177 thread-budget accounting.

Existing implementation retained

Preserve current star.runtime, consumer owned-thread retention, finite Rabbit polling, Clack handle stop, Sento shutdown :wait t, lparallel shutdown, signal handling, startup rollback and live/ready endpoints.

This is a refinement/migration of hard-coded orchestration into registered components, not a rewrite.

Common Lisp component contract

Conceptual protocol:

register-runtime-component(runtime, component)
component-id(component) -> stable symbol/id
component-dependencies(component) -> ids
component-required-p(component) -> boolean
component-state(component) -> state
component-start(component, context) -> resources | failure
component-quiesce(component, deadline) -> phase-result
component-drain(component, deadline) -> phase-result
component-stop(component, deadline) -> phase-result
component-ready-p(component) -> boolean + reason
component-reconnect-policy(component) -> policy | none

Use ordinary Common Lisp protocol/generic functions or structs/classes consistent with repository style. No StarLang grammar is added.

Registration must reject:

duplicate component IDs;
unknown dependencies;
dependency cycles;
owned resources/threads with no component owner;
a reconnect policy on a component that has no reconnect adapter.

Initial component inventory

Migrate current hard-coded owners behind component adapters, in dependency order where applicable:

database/bootstrap/auth initialization resources where cleanup exists;
lparallel CPU kernel;
Sento actor system + StarIntel actor-owned timers/agents;
Rabbit producer/owner resources;
Rabbit consumer groups;
event consumer;
HTTP/Clack server;
later #35 workload dispatchers/router pools;
later #36 admission/drain boundaries;
KV/CouchDB/Rabbit reconnectable service adapters as appropriate.

Do not pretend pure initialization with no retained resource is a lifecycle component unless it needs cleanup/readiness/reconnect semantics.

Startup algorithm

validate component graph;
validate #177 thread-demand plan before starting long-lived resources;
topologically order components;
for each component:
- transition registered -> starting;
- invoke start;
- retain resources and owner identity;
- transition to running only after successful start contract;
if any component fails:
- freeze further starts;
- record startup failure;
- stop only the successfully started prefix in reverse dependency order;
- return startup failure plus rollback report;
- never hide rollback failures.

Start/stop remains idempotent. A second start of an active runtime rejects; stop of absent/stopped runtime succeeds without side effects.

One absolute shutdown deadline

Use one process-wide shutdown duration authority (current *shutdown-timeout-seconds* may remain the source value). At stop initiation compute a monotonic absolute deadline once.

Every phase receives the same absolute deadline or a context exposing remaining-time. Components never receive a fresh full timeout.

Required invariant:

end_time(stop-runtime) <= shutdown_start + configured_shutdown_budget + tiny scheduler/test tolerance

A component that exhausts the remaining budget returns timedOut. The runtime continues only with cleanup operations that are themselves bounded by remaining time or safe immediate fencing.

Shutdown phases

1. Quiesce

Stop new external/data-plane intake while retaining control capacity.

Examples:

HTTP stops accepting new mutating/bulk work or closes listener according to adapter capability;
Rabbit consumers stop pulling new deliveries;
target/research dispatch stops creating new work.

#36 defines overload/rejection semantics. #37 merely invokes the lifecycle boundary.

2. Drain

Wait for already accepted work within the remaining deadline.

Each component reports counts where meaningful:

acceptedBeforeQuiesce
completedDuringDrain
rejectedAfterQuiesce
remainingAtDeadline
handoffOrDurableState

No component may wait forever.

3. Stop

Release resources in reverse dependency order: consumers/workers, actors/timers, transport connections, executors, etc., as dictated by graph dependencies rather than a second handwritten shutdown list.

Shutdown result

Do not collapse everything to boolean t.

Conceptual result:

ShutdownReport {
  reason
  startedAt
  deadline
  completedAt
  outcome: clean | incomplete | timedOut
  componentResults[]
}

ComponentStopResult {
  componentId
  quiesceResult
  drainResult
  stopResult
  remainingOwnedResources
  errors[]
}

The runtime may enter :stopped as a lifecycle fact after bounded teardown is over, but the report must preserve whether cleanup was clean. Readiness is false from the moment quiesce/stopping begins.

Reconnect supervisor

Reconnect is a component lifecycle sub-state, not a detached worker architecture.

States:

running -> degraded -> reconnecting -> running
                          |              |
                          +-> failed <---+

Rules:

stable component ID across attempts;
monotonically increasing reconnect generation;
attempt N may install resources only if its generation is still current and runtime is still running;
quiesce/stop increments/fences generation and cancels pending backoff;
late result from stale generation is closed/discarded, never installed;
exponential backoff with bounded jitter/cap and bounded attempt/window policy;
no reconnect after terminal runtime stop;
readiness false for required component while degraded/reconnecting;
reconnect cannot duplicate consumers, producer channels, or service connections.

Rabbit, CouchDB and KV adapters opt into this independently. Do not force identical reconnection mechanics below the lifecycle protocol.

Health/readiness contract

Replace handle-presence assumptions with component probes.

Runtime readiness requires:

runtime state :running;
all required components state running;
every required component readiness probe true;
no required component in degraded/reconnecting/failed state;
#177 owned-thread/accounting validation true once integrated.

Optional component failure may produce degraded diagnostics without making the whole runtime unready only when the component is explicitly declared optional.

#177 thread-budget integration

Every lifecycle component has the stable owner ID used by thread-demand allocations and live thread registration.

Invariant:

runtime-owned-thread -> exactly one componentId -> zero/one allocation record

Thread-producing components must obtain the relevant grant before realizing workers. #37 does not duplicate the budget planner.

#35/#36 boundaries

#35 dispatcher/router components register lifecycle ownership and consume #177 grants.
#36 admission components expose quiesce/drain status and derive queue/prefetch policy from granted capacity.
#37 owns orchestration timing/order, not workload routing policy or overload decisions.

StarLang boundary

No new syntax/runtime extension is required for this host-level process lifecycle problem.

StarLang actors/domain servers continue to use their lifecycle/cancellation semantics inside their host. The Common Lisp host is responsible for process resources, OS threads, Clack, Sento, Rabbit, CouchDB/KV adapters and their lifecycle ownership. No Python/external supervisor is justified.

Migration slices

All slices remain AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL.

Component registry + graph validation + pure ordering
- introduce component model under existing star.runtime;
- no production component migration yet.
Absolute deadline + structured shutdown report
- refactor current stop path to one deadline;
- preserve existing concrete stop adapters.
Migrate current hard-coded components into registry
- actor system/timers, consumers/event consumer, HTTP, lparallel, producer/transport resources.
Explicit quiesce/drain contract
- connect Rabbit/HTTP/accepted-work accounting without implementing #36's full admission policy.
Reconnect generation/fencing + one reconnectable adapter
- prove with Rabbit or a fake service first, then CouchDB/KV adapters in bounded follow-ups.
Readiness from component health + #177 owned-resource accounting
- remove handle-presence shortcuts.
#35/#36 integration
- register new pools/admission boundaries as components rather than adding special cases.

Mandatory RED-first tests

Slice 1

Before production mutation, add pure fixtures:

A -> B -> C yields deterministic topological start and reverse stop order;
dependency cycle is rejected;
unknown dependency is rejected;
duplicate component ID is rejected.

Untouched current code must fail because no component registry/graph exists.

Slice 2

Fake three stop adapters that would each consume a full current timeout. Assert the runtime passes one shared deadline and total stop cannot consume three fresh budgets. Untouched code must fail this contract.

Also assert a stop failure appears in ShutdownReport and does not disappear behind ignore-errors.

Slice 3

Start failure at component N must stop exactly the successfully started prefix once, reverse ordered, with no stop call for never-started components.

Slice 4

A fake intake component must reject new work after quiesce while previously accepted work can finish until the shared deadline.

Slice 5

Reconnect generation N completes after N+1 and after runtime stop. Both stale completions must be fenced and their resources closed/not installed.

Slice 6

A retained non-NIL component handle whose explicit health probe is false must make runtime readiness false.

Adversarial review

Rejected:

a new process manager alongside star.runtime;
a generic third-party service framework introduced merely to avoid a small CL protocol;
fresh timeout per phase/component;
cleanup failure swallowed while reporting clean shutdown;
detached reconnect threads or timers without component ownership;
reconnect during stopping;
force termination before bounded graceful drain;
querying bt:all-threads for ownership;
making resource count/health inferred from object non-NILness;
letting component adapters size their own pools, bypassing #177;
pushing process lifecycle semantics into StarLang;
broad rewrite of current lifecycle code before pure graph/deadline contracts are RED.

Acceptance

every retained runtime resource has one lifecycle component owner;
startup order derives from a validated dependency graph;
partial startup rollback is deterministic and observable;
process shutdown obeys one absolute bounded deadline;
shutdown report distinguishes clean/incomplete/timed-out teardown;
no unrelated process thread is joined;
required-component health drives readiness;
reconnect is bounded, fenced, cancellation-aware and cannot resurrect stopped components;
#177 thread allocations/accounting attach to lifecycle owner IDs;
#35/#36 can integrate without adding another lifecycle control plane;
no new StarLang syntax or external process supervisor.

Implementation approval

PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL

Only the operator may authorize implementation.

## Authority / state - Source research: #182 - Owning implementation issue: `lost-rob0t/starintel-server#37` - Related P0 design: #177 thread-budget authority - Related P0 issues: `starintel-server#35`, `#36` - Current server master inspected: `8fb297d146e7332fae7e38170b5b49d49530ac53` - Research state: `READY_FOR_DESIGN` - Design state: `DESIGN_READY_FOR_OPERATOR_REVIEW` - **Implementation approval: PENDING / AWAITING_OPERATOR_APPROVAL — operator only** This design must not enter executable RAGE until explicit operator implementation approval. ## Design goal Extend the **existing** `star.runtime` lifecycle owner into a closed component/resource supervisor. Do not replace it and do not introduce a second service manager. The resulting runtime must: 1. know every owned component/resource/thread; 2. start in dependency order and roll back deterministically; 3. quiesce/drain/stop under one absolute process shutdown deadline; 4. report incomplete cleanup rather than silently calling it clean; 5. own bounded reconnect for reconnectable dependencies; 6. expose readiness from component health rather than retained handle presence; 7. provide the owner identity consumed by #177 thread-budget accounting. ## Existing implementation retained Preserve current `star.runtime`, consumer owned-thread retention, finite Rabbit polling, Clack handle stop, Sento `shutdown :wait t`, lparallel shutdown, signal handling, startup rollback and live/ready endpoints. This is a refinement/migration of hard-coded orchestration into registered components, not a rewrite. ## Common Lisp component contract Conceptual protocol: ```text register-runtime-component(runtime, component) component-id(component) -> stable symbol/id component-dependencies(component) -> ids component-required-p(component) -> boolean component-state(component) -> state component-start(component, context) -> resources | failure component-quiesce(component, deadline) -> phase-result component-drain(component, deadline) -> phase-result component-stop(component, deadline) -> phase-result component-ready-p(component) -> boolean + reason component-reconnect-policy(component) -> policy | none ``` Use ordinary Common Lisp protocol/generic functions or structs/classes consistent with repository style. No StarLang grammar is added. Registration must reject: - duplicate component IDs; - unknown dependencies; - dependency cycles; - owned resources/threads with no component owner; - a reconnect policy on a component that has no reconnect adapter. ## Initial component inventory Migrate current hard-coded owners behind component adapters, in dependency order where applicable: - database/bootstrap/auth initialization resources where cleanup exists; - lparallel CPU kernel; - Sento actor system + StarIntel actor-owned timers/agents; - Rabbit producer/owner resources; - Rabbit consumer groups; - event consumer; - HTTP/Clack server; - later #35 workload dispatchers/router pools; - later #36 admission/drain boundaries; - KV/CouchDB/Rabbit reconnectable service adapters as appropriate. Do not pretend pure initialization with no retained resource is a lifecycle component unless it needs cleanup/readiness/reconnect semantics. ## Startup algorithm 1. validate component graph; 2. validate #177 thread-demand plan before starting long-lived resources; 3. topologically order components; 4. for each component: - transition `registered -> starting`; - invoke start; - retain resources and owner identity; - transition to `running` only after successful start contract; 5. if any component fails: - freeze further starts; - record startup failure; - stop only the successfully started prefix in reverse dependency order; - return startup failure plus rollback report; - never hide rollback failures. Start/stop remains idempotent. A second start of an active runtime rejects; stop of absent/stopped runtime succeeds without side effects. ## One absolute shutdown deadline Use one process-wide shutdown duration authority (current `*shutdown-timeout-seconds*` may remain the source value). At stop initiation compute a monotonic absolute deadline once. Every phase receives the same absolute deadline or a context exposing `remaining-time`. Components never receive a fresh full timeout. Required invariant: ```text end_time(stop-runtime) <= shutdown_start + configured_shutdown_budget + tiny scheduler/test tolerance ``` A component that exhausts the remaining budget returns `timedOut`. The runtime continues only with cleanup operations that are themselves bounded by remaining time or safe immediate fencing. ## Shutdown phases ### 1. Quiesce Stop new external/data-plane intake while retaining control capacity. Examples: - HTTP stops accepting new mutating/bulk work or closes listener according to adapter capability; - Rabbit consumers stop pulling new deliveries; - target/research dispatch stops creating new work. #36 defines overload/rejection semantics. #37 merely invokes the lifecycle boundary. ### 2. Drain Wait for already accepted work within the remaining deadline. Each component reports counts where meaningful: ```text acceptedBeforeQuiesce completedDuringDrain rejectedAfterQuiesce remainingAtDeadline handoffOrDurableState ``` No component may wait forever. ### 3. Stop Release resources in reverse dependency order: consumers/workers, actors/timers, transport connections, executors, etc., as dictated by graph dependencies rather than a second handwritten shutdown list. ## Shutdown result Do not collapse everything to boolean `t`. Conceptual result: ```text ShutdownReport { reason startedAt deadline completedAt outcome: clean | incomplete | timedOut componentResults[] } ComponentStopResult { componentId quiesceResult drainResult stopResult remainingOwnedResources errors[] } ``` The runtime may enter `:stopped` as a lifecycle fact after bounded teardown is over, but the report must preserve whether cleanup was clean. Readiness is false from the moment quiesce/stopping begins. ## Reconnect supervisor Reconnect is a component lifecycle sub-state, not a detached worker architecture. States: ```text running -> degraded -> reconnecting -> running | | +-> failed <---+ ``` Rules: - stable component ID across attempts; - monotonically increasing reconnect generation; - attempt N may install resources only if its generation is still current and runtime is still running; - quiesce/stop increments/fences generation and cancels pending backoff; - late result from stale generation is closed/discarded, never installed; - exponential backoff with bounded jitter/cap and bounded attempt/window policy; - no reconnect after terminal runtime stop; - readiness false for required component while degraded/reconnecting; - reconnect cannot duplicate consumers, producer channels, or service connections. Rabbit, CouchDB and KV adapters opt into this independently. Do not force identical reconnection mechanics below the lifecycle protocol. ## Health/readiness contract Replace handle-presence assumptions with component probes. Runtime readiness requires: - runtime state `:running`; - all required components state `running`; - every required component readiness probe true; - no required component in degraded/reconnecting/failed state; - #177 owned-thread/accounting validation true once integrated. Optional component failure may produce degraded diagnostics without making the whole runtime unready only when the component is explicitly declared optional. ## #177 thread-budget integration Every lifecycle component has the stable owner ID used by thread-demand allocations and live thread registration. Invariant: ```text runtime-owned-thread -> exactly one componentId -> zero/one allocation record ``` Thread-producing components must obtain the relevant grant before realizing workers. #37 does not duplicate the budget planner. ## #35/#36 boundaries - #35 dispatcher/router components register lifecycle ownership and consume #177 grants. - #36 admission components expose quiesce/drain status and derive queue/prefetch policy from granted capacity. - #37 owns orchestration timing/order, not workload routing policy or overload decisions. ## StarLang boundary No new syntax/runtime extension is required for this host-level process lifecycle problem. StarLang actors/domain servers continue to use their lifecycle/cancellation semantics inside their host. The Common Lisp host is responsible for process resources, OS threads, Clack, Sento, Rabbit, CouchDB/KV adapters and their lifecycle ownership. No Python/external supervisor is justified. ## Migration slices All slices remain **AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL**. 1. **Component registry + graph validation + pure ordering** - introduce component model under existing `star.runtime`; - no production component migration yet. 2. **Absolute deadline + structured shutdown report** - refactor current stop path to one deadline; - preserve existing concrete stop adapters. 3. **Migrate current hard-coded components into registry** - actor system/timers, consumers/event consumer, HTTP, lparallel, producer/transport resources. 4. **Explicit quiesce/drain contract** - connect Rabbit/HTTP/accepted-work accounting without implementing #36's full admission policy. 5. **Reconnect generation/fencing + one reconnectable adapter** - prove with Rabbit or a fake service first, then CouchDB/KV adapters in bounded follow-ups. 6. **Readiness from component health + #177 owned-resource accounting** - remove handle-presence shortcuts. 7. **#35/#36 integration** - register new pools/admission boundaries as components rather than adding special cases. ## Mandatory RED-first tests ### Slice 1 Before production mutation, add pure fixtures: - A -> B -> C yields deterministic topological start and reverse stop order; - dependency cycle is rejected; - unknown dependency is rejected; - duplicate component ID is rejected. Untouched current code must fail because no component registry/graph exists. ### Slice 2 Fake three stop adapters that would each consume a full current timeout. Assert the runtime passes one shared deadline and total stop cannot consume three fresh budgets. Untouched code must fail this contract. Also assert a stop failure appears in `ShutdownReport` and does not disappear behind `ignore-errors`. ### Slice 3 Start failure at component N must stop exactly the successfully started prefix once, reverse ordered, with no stop call for never-started components. ### Slice 4 A fake intake component must reject new work after quiesce while previously accepted work can finish until the shared deadline. ### Slice 5 Reconnect generation N completes after N+1 and after runtime stop. Both stale completions must be fenced and their resources closed/not installed. ### Slice 6 A retained non-NIL component handle whose explicit health probe is false must make runtime readiness false. ## Adversarial review Rejected: 1. a new process manager alongside `star.runtime`; 2. a generic third-party service framework introduced merely to avoid a small CL protocol; 3. fresh timeout per phase/component; 4. cleanup failure swallowed while reporting clean shutdown; 5. detached reconnect threads or timers without component ownership; 6. reconnect during stopping; 7. force termination before bounded graceful drain; 8. querying `bt:all-threads` for ownership; 9. making resource count/health inferred from object non-NILness; 10. letting component adapters size their own pools, bypassing #177; 11. pushing process lifecycle semantics into StarLang; 12. broad rewrite of current lifecycle code before pure graph/deadline contracts are RED. ## Acceptance - every retained runtime resource has one lifecycle component owner; - startup order derives from a validated dependency graph; - partial startup rollback is deterministic and observable; - process shutdown obeys one absolute bounded deadline; - shutdown report distinguishes clean/incomplete/timed-out teardown; - no unrelated process thread is joined; - required-component health drives readiness; - reconnect is bounded, fenced, cancellation-aware and cannot resurrect stopped components; - #177 thread allocations/accounting attach to lifecycle owner IDs; - #35/#36 can integrate without adding another lifecycle control plane; - no new StarLang syntax or external process supervisor. ## Implementation approval `PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL` Only the operator may authorize implementation.