[P0-29] Implement managed startup, graceful shutdown, reconnect, and owned-thread cleanup #37

New issue

Open

opened 2026-07-22 00:15:46 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-07-22 00:15:46 +00:00

(Migrated from github.com)

Governance / ARDR state

Research authority: lost-rob0t/starintel-auto-research#182 — READY_FOR_DESIGN
Design authority: lost-rob0t/starintel-auto-research#183 — DESIGN_READY_FOR_OPERATOR_REVIEW
Related P0 runtime authority: #34 / auto-research #177 thread budget
Related P0 follow-ons: #35 workload dispatchers, #36 admission/overload
Current master inspected: 8fb297d146e7332fae7e38170b5b49d49530ac53
Implementation approval: PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL

This issue must not enter executable RAGE until explicit operator implementation approval.

Current state: original defect partially fixed

Current master already has source/runtime-lifecycle.lisp and lifecycle tests. Preserve:

existing star.runtime authority and lifecycle states;
retained Rabbit consumer thread refs and owned-thread-only joins;
finite consume polling for stop observation;
retained Clack handle and bounded stop;
Sento actor-system shutdown :wait t;
lparallel kernel shutdown;
startup rollback;
SIGTERM/SIGINT handling;
live/readiness endpoints;
regression proving unrelated process threads are not joined.

Do not implement a second runtime owner or rewrite this foundation.

Remaining P0 defect

The runtime still hard-codes component ordering/slots, repeatedly starts fresh timeout windows during shutdown, can swallow cleanup failures, has no closed quiesce/drain component protocol, and has no bounded reconnect/fencing owner for Rabbit/CouchDB/KV dependencies. Some readiness checks still infer health from retained handles.

Approved architecture status

Design is ready for operator review only. No implementation slice is approved yet.

The proposed architecture extends star.runtime with a closed Common Lisp runtime-component registry/protocol. No new StarLang syntax and no external/Python supervisor.

Every component has:

stable owner ID;
dependency IDs;
required/optional classification;
lifecycle state;
owned resources/thread-allocation refs;
start/quiesce/drain/stop operations;
explicit readiness probe;
optional reconnect policy.

Startup derives deterministic topological order. Rollback stops exactly the successfully started prefix in reverse dependency order.

Shutdown computes one monotonic absolute deadline once. Every quiesce/drain/stop adapter consumes remaining time from that deadline; no phase gets a fresh full timeout.

Reconnect uses bounded backoff plus monotonically increasing generation/fencing. A late reconnect completion may install resources only if its generation is still current and the runtime is still running. Stop fences pending attempts so they cannot resurrect resources.

Dependency-ordered proposed implementation slices

All are AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL.

P0 / prerequisite — component registry + graph validation
- pure component model under existing star.runtime;
- deterministic topological start/reverse stop order;
- reject duplicate IDs, unknown dependencies and cycles;
- no production component migration in this slice.
P0 — one absolute shutdown deadline + structured report
- replace repeated fresh timeout windows with one process deadline;
- preserve current concrete stop adapters initially;
- return clean/incomplete/timed-out shutdown outcome with per-component failures.
P0 — migrate current hard-coded resources into component registry
- Sento actor system/timers/agents;
- Rabbit producer/consumers/event consumer;
- Clack HTTP server;
- lparallel kernel;
- attach #177 thread-allocation owner IDs.
P0 — quiesce/drain lifecycle phase
- stop new intake before teardown;
- permit accepted work to finish only within remaining process deadline;
- expose drain/rejected/remaining counts where meaningful;
- #36 remains authority for overload/admission policy.
P0 — reconnect generation/fencing
- prove generic protocol with fake adapter, then one real reconnectable dependency;
- bounded exponential backoff/jitter/cap/attempt window;
- no reconnect while quiescing/stopping/stopped;
- stale completion closes/discards its resource.
P0 — component health readiness + owned-resource accounting
- required component explicit health drives readiness;
- non-NIL handle is insufficient;
- integrate #177 live thread/resource ownership validation.
Follow-on integration
- #35 dispatchers/router pools register as components and consume thread grants;
- #36 admission/drain boundaries register without another lifecycle control plane.

Mandatory RED-first gate

No production mutation before the selected slice has legitimate RED evidence on untouched baseline.

First RED for slice 1

Add pure deterministic fixtures requiring a runtime-component graph:

A -> B -> C yields start order A B C and stop order C B A;
dependency cycle is rejected;
unknown dependency is rejected;
duplicate component ID is rejected.

Untouched current master must fail because star.runtime has no component registry/graph protocol.

RED for slice 2

Fake three stop components whose adapters would each consume a fresh current timeout. Assert one shared absolute deadline is propagated and total shutdown cannot consume three fresh budgets. Untouched current lifecycle must fail this contract.

Also assert cleanup failure is retained in a structured shutdown result rather than disappearing behind ignore-errors/logging.

RED for reconnect slice

Reconnect generation N completes after N+1 and after runtime stop. Both stale completions must be rejected, must not replace current resources, and must close/fence their returned resource.

RED for readiness/accounting slice

retained non-NIL handle + failed explicit component health => runtime unready;
runtime-owned resource/thread with no registered component owner => validation failure.

Acceptance criteria

exactly one existing star.runtime authority owns lifecycle;
every retained runtime resource/thread maps to one component owner;
dependency graph determines startup/rollback/shutdown ordering;
partial startup rollback is exact, reverse ordered and observable;
shutdown obeys one absolute configured deadline rather than N repeated timeouts;
cleanup failures are reported as incomplete/timed-out rather than silently clean;
runtime never joins unrelated process threads;
quiesce prevents new intake and drain is bounded;
reconnect is bounded, cancellation-aware and generation-fenced;
stopped runtime cannot be resurrected by stale reconnect completion;
required component health drives readiness;
#177 thread-budget grants/accounting use the same owner IDs;
#35/#36 integrate without a second lifecycle or capacity authority;
no new StarLang syntax, external supervisor or Python controller.

Implementation approval

PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL

ARDR/ADARD cannot change this state. Explicit operator approval is required before RAGE implementation.

## Governance / ARDR state - Research authority: `lost-rob0t/starintel-auto-research#182` — `READY_FOR_DESIGN` - Design authority: `lost-rob0t/starintel-auto-research#183` — `DESIGN_READY_FOR_OPERATOR_REVIEW` - Related P0 runtime authority: #34 / auto-research #177 thread budget - Related P0 follow-ons: #35 workload dispatchers, #36 admission/overload - Current master inspected: `8fb297d146e7332fae7e38170b5b49d49530ac53` - **Implementation approval: PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL** This issue must not enter executable RAGE until explicit operator implementation approval. ## Current state: original defect partially fixed Current master already has `source/runtime-lifecycle.lisp` and lifecycle tests. Preserve: - existing `star.runtime` authority and lifecycle states; - retained Rabbit consumer thread refs and owned-thread-only joins; - finite consume polling for stop observation; - retained Clack handle and bounded stop; - Sento actor-system `shutdown :wait t`; - lparallel kernel shutdown; - startup rollback; - SIGTERM/SIGINT handling; - live/readiness endpoints; - regression proving unrelated process threads are not joined. Do **not** implement a second runtime owner or rewrite this foundation. ## Remaining P0 defect The runtime still hard-codes component ordering/slots, repeatedly starts fresh timeout windows during shutdown, can swallow cleanup failures, has no closed quiesce/drain component protocol, and has no bounded reconnect/fencing owner for Rabbit/CouchDB/KV dependencies. Some readiness checks still infer health from retained handles. ## Approved architecture status Design is ready for operator review only. **No implementation slice is approved yet.** The proposed architecture extends `star.runtime` with a closed Common Lisp runtime-component registry/protocol. No new StarLang syntax and no external/Python supervisor. Every component has: - stable owner ID; - dependency IDs; - required/optional classification; - lifecycle state; - owned resources/thread-allocation refs; - start/quiesce/drain/stop operations; - explicit readiness probe; - optional reconnect policy. Startup derives deterministic topological order. Rollback stops exactly the successfully started prefix in reverse dependency order. Shutdown computes **one monotonic absolute deadline once**. Every quiesce/drain/stop adapter consumes remaining time from that deadline; no phase gets a fresh full timeout. Reconnect uses bounded backoff plus monotonically increasing generation/fencing. A late reconnect completion may install resources only if its generation is still current and the runtime is still running. Stop fences pending attempts so they cannot resurrect resources. ## Dependency-ordered proposed implementation slices All are **AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL**. 1. **P0 / prerequisite — component registry + graph validation** - pure component model under existing `star.runtime`; - deterministic topological start/reverse stop order; - reject duplicate IDs, unknown dependencies and cycles; - no production component migration in this slice. 2. **P0 — one absolute shutdown deadline + structured report** - replace repeated fresh timeout windows with one process deadline; - preserve current concrete stop adapters initially; - return clean/incomplete/timed-out shutdown outcome with per-component failures. 3. **P0 — migrate current hard-coded resources into component registry** - Sento actor system/timers/agents; - Rabbit producer/consumers/event consumer; - Clack HTTP server; - lparallel kernel; - attach #177 thread-allocation owner IDs. 4. **P0 — quiesce/drain lifecycle phase** - stop new intake before teardown; - permit accepted work to finish only within remaining process deadline; - expose drain/rejected/remaining counts where meaningful; - #36 remains authority for overload/admission policy. 5. **P0 — reconnect generation/fencing** - prove generic protocol with fake adapter, then one real reconnectable dependency; - bounded exponential backoff/jitter/cap/attempt window; - no reconnect while quiescing/stopping/stopped; - stale completion closes/discards its resource. 6. **P0 — component health readiness + owned-resource accounting** - required component explicit health drives readiness; - non-NIL handle is insufficient; - integrate #177 live thread/resource ownership validation. 7. **Follow-on integration** - #35 dispatchers/router pools register as components and consume thread grants; - #36 admission/drain boundaries register without another lifecycle control plane. ## Mandatory RED-first gate No production mutation before the selected slice has legitimate RED evidence on untouched baseline. ### First RED for slice 1 Add pure deterministic fixtures requiring a runtime-component graph: - `A -> B -> C` yields start order `A B C` and stop order `C B A`; - dependency cycle is rejected; - unknown dependency is rejected; - duplicate component ID is rejected. Untouched current master must fail because `star.runtime` has no component registry/graph protocol. ### RED for slice 2 Fake three stop components whose adapters would each consume a fresh current timeout. Assert one shared absolute deadline is propagated and total shutdown cannot consume three fresh budgets. Untouched current lifecycle must fail this contract. Also assert cleanup failure is retained in a structured shutdown result rather than disappearing behind `ignore-errors`/logging. ### RED for reconnect slice Reconnect generation N completes after N+1 and after runtime stop. Both stale completions must be rejected, must not replace current resources, and must close/fence their returned resource. ### RED for readiness/accounting slice - retained non-NIL handle + failed explicit component health => runtime unready; - runtime-owned resource/thread with no registered component owner => validation failure. ## Acceptance criteria - exactly one existing `star.runtime` authority owns lifecycle; - every retained runtime resource/thread maps to one component owner; - dependency graph determines startup/rollback/shutdown ordering; - partial startup rollback is exact, reverse ordered and observable; - shutdown obeys one absolute configured deadline rather than N repeated timeouts; - cleanup failures are reported as incomplete/timed-out rather than silently clean; - runtime never joins unrelated process threads; - quiesce prevents new intake and drain is bounded; - reconnect is bounded, cancellation-aware and generation-fenced; - stopped runtime cannot be resurrected by stale reconnect completion; - required component health drives readiness; - #177 thread-budget grants/accounting use the same owner IDs; - #35/#36 integrate without a second lifecycle or capacity authority; - no new StarLang syntax, external supervisor or Python controller. ## Implementation approval `PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL` ARDR/ADARD cannot change this state. Explicit operator approval is required before RAGE implementation.