nsaspy/starintel-auto-research

Fork 0

P0 ARDR research: finish managed runtime lifecycle, deadlines, and reconnect supervision #201

New issue

Open

opened 2026-09-02 17:26:16 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-02 17:26:16 +00:00

Owner

StarIntel problem solved

Research the remaining P0 lifecycle gap in lost-rob0t/starintel-server#37 against current source rather than the stale original issue description.

Owning implementation issue: lost-rob0t/starintel-server#37.
Related P0 runtime work: #34 thread-budget authority, #35 workload dispatchers, #36 admission/overload.
Current starintel-server/master inspected at 8fb297d146e7332fae7e38170b5b49d49530ac53.

ARDR state

READY_FOR_DESIGN

Implementation approval remains operator-only.

Candidate/ranking decision

This is P0, concrete, and dependency-unblocking. #34 already has research/design authority; #35 and #36 depend on trustworthy runtime ownership and shutdown semantics. #37 therefore wins this bounded cycle over lower-priority datasource/actor enhancement work.

Current implementation check

The original #37 defect list is partially stale. Current master already contains source/runtime-lifecycle.lisp and t/runtime-lifecycle-test.lisp.

Already implemented and to be preserved:

a star.runtime owner with :created/:starting/:running/:stopping/:stopped state;
retained runtime handles for actor system, Rabbit consumers, event consumer, HTTP server, and lparallel kernel;
consumer worker thread references and owned-thread-only joins;
finite Rabbit consume polling so workers can observe stop state;
Clack HTTP handle retention and clack:stop shutdown;
Sento actor-system shutdown with :wait t;
lparallel kernel shutdown;
startup rollback through stop-runtime on startup failure;
SIGTERM/SIGINT stop request handling;
/live, /ready, /health lifecycle endpoints;
tests proving consumer shutdown does not join an unrelated thread and readiness fails closed on Rabbit/CouchDB probe failure.

This means ARDR must not propose a second runtime owner.

Residual correctness gaps

1. Runtime ownership is still hard-coded, not closed

star-runtime has dedicated slots and start-runtime/stop-runtime directly know each subsystem. New thread/resource-producing components can therefore be added without registering lifecycle ownership, start dependency, stop order, health, or cleanup behavior.

Requirement: one closed component/resource registration contract under the existing Common Lisp star.runtime authority. Do not create another supervisor/control plane.

2. One timeout is repeatedly restarted instead of one shutdown deadline

Current stop phases each receive or establish the same timeout independently. Total process shutdown can therefore exceed the operator's intended bound by roughly N phases.

Requirement: compute one monotonic absolute shutdown deadline at stop initiation and pass remaining time to every phase/component. Once exhausted, later phases may perform only bounded emergency cleanup/diagnostics; they may not restart the full timeout.

3. Cleanup failures are weakly observable

Several stop paths use ignore-errors or log-and-continue. stop-runtime can reach :stopped even when a component failed to stop. This loses the distinction between clean shutdown and bounded but incomplete cleanup.

Requirement: aggregate typed per-component stop results and terminal shutdown outcome. Preserve cleanup failure evidence while continuing best-effort teardown.

4. Reconnect/restart supervision is absent

Current Rabbit consumer loop exits on non-timeout transport error; CouchDB readiness is a probe; producer readiness is checked, but there is no runtime-owned reconnect/backoff state machine for Rabbit, CouchDB, or KV dependencies.

Requirement: reconnect belongs to the existing runtime/component owner. Use bounded exponential backoff with jitter/cap, cancellation on runtime stop, stable component identity, and readiness degradation while disconnected. Do not spawn detached reconnect threads.

5. Readiness is not equivalent to owned-component health

Actor/kernel/HTTP readiness is primarily non-NIL handle/state presence. A dead internal worker or failed component can exist behind a retained handle.

Requirement: component readiness must be an explicit probe/status from the owning adapter. Runtime readiness is the conjunction of required component states, not object existence.

6. No explicit drain contract

Stopping HTTP and consumers prevents new intake, but current lifecycle has no general quiesce -> drain -> stop component contract, no shared remaining-deadline propagation, and no structured accounting of drained/rejected/handoff work.

Requirement: v1 lifecycle phases are start, quiesce, drain, stop; components that have no drain work may no-op. #36 owns admission/overload policy; #37 owns when intake closes and how drain deadline propagates.

7. Thread-budget accounting must attach to lifecycle ownership

#177 requires every runtime-owned thread to map to a component/allocation. Lifecycle registration is the natural owner identity. A component may own zero or more thread allocations/resources, but raw product-side thread creation cannot bypass both registries.

Evidence-backed library boundary

Current upstream Sento exposes actor/context/system shutdown with an optional wait for complete shutdown and dispatcher stop semantics. Keep using that as the actor adapter rather than reimplementing actor teardown.

Current Clack exposes server shutdown through the retained handler returned by clackup; keep this handle as the HTTP component resource.

Bordeaux Threads is the portability layer for thread ownership/joining, not a lifecycle registry. The runtime must therefore retain exact thread/resource ownership rather than enumerate process threads.

Required lifecycle model

Conceptual Common Lisp model, not new StarLang syntax:

RuntimeComponent {
  id
  dependencies[]
  required?
  state
  resources[]
  threadAllocationRefs[]
  start
  quiesce
  drain
  stop
  readiness
  reconnectPolicy?
}

ComponentState =
  registered | starting | running | degraded | reconnecting |
  quiescing | draining | stopping | stopped | failed

Startup performs a deterministic topological order. Rollback stops only components that successfully crossed into owned started state, in reverse dependency order.

Shutdown computes one absolute deadline and executes reverse dependency order through quiesce/drain/stop while accumulating outcomes.

Reconnect semantics

Reconnect applies only to components whose adapter declares it and never runs once runtime state is quiescing/stopping/stopped.

Minimum policy:

bounded exponential backoff;
configurable cap and maximum consecutive attempts/window;
jitter to avoid synchronized reconnect storms;
reset after stable healthy period;
readiness false/degraded during reconnect;
no duplicate consumer/producer ownership after reconnect;
reconnect generation/fence so late completion from an older attempt cannot replace the newer resource;
stop cancels pending backoff and fences late reconnect completion.

StarLang classification

No new StarLang syntax is justified.

This is host process lifecycle and resource ownership in starintel-server. StarLang actors/domain servers may expose lifecycle/health through their existing runtime boundary, but cannot own process thread/transport authority. Implement in Common Lisp using current libraries. No external process or Python controller is justified.

Adversarial findings

Rejected:

Rewrite the existing runtime owner. Current implementation already supplies the right authority boundary; extend it.
One timeout per stop phase. Violates the process shutdown bound.
Join bt:all-threads. Can block on or interfere with threads StarIntel does not own.
Mark :stopped after swallowed cleanup errors with no result. Loses operational truth.
Detached reconnect loops. Create new unowned resources and can resurrect components during shutdown.
Infinite reconnect. Converts dependency outage into uncontrolled background work.
Readiness = non-NIL handle. Cannot detect a dead worker behind a retained object.
Force-kill first. Risks lost Rabbit settlements/outbox/lease state; graceful bounded drain comes first.
Make #37 own overload policy. #36 owns admission decisions; #37 owns lifecycle transition and deadline propagation.
New StarLang lifecycle grammar. No language-level semantic gap is demonstrated.

Mandatory RED-first targets

Absolute deadline fixture: fake three components whose individual stop calls each consume part of a shared deadline. Untouched current runtime should fail because it restarts timeout windows rather than propagating one absolute deadline.
Partial rollback fixture: component B start fails after A starts; prove only successfully started components are stopped exactly once in reverse dependency order and cleanup failures remain in the returned shutdown report.
Reconnect fencing fixture: reconnect attempt generation N completes after N+1 or after stop; stale completion must be rejected and may not install a resource.
Readiness fixture: retained component handle with failed explicit health status must make runtime unready.
Ownership fixture: a runtime-owned thread/resource without a registered component owner must fail lifecycle/accounting validation.

Decision

Evidence is sufficient to proceed directly to design. The architecture boundary is clear and current implementation provides the seed rather than requiring a rewrite.

Implementation approval: PENDING / AWAITING_OPERATOR_APPROVAL.

## StarIntel problem solved Research the **remaining** P0 lifecycle gap in `lost-rob0t/starintel-server#37` against current source rather than the stale original issue description. Owning implementation issue: `lost-rob0t/starintel-server#37`. Related P0 runtime work: #34 thread-budget authority, #35 workload dispatchers, #36 admission/overload. Current `starintel-server/master` inspected at `8fb297d146e7332fae7e38170b5b49d49530ac53`. ## ARDR state `READY_FOR_DESIGN` Implementation approval remains operator-only. ## Candidate/ranking decision This is P0, concrete, and dependency-unblocking. #34 already has research/design authority; #35 and #36 depend on trustworthy runtime ownership and shutdown semantics. #37 therefore wins this bounded cycle over lower-priority datasource/actor enhancement work. ## Current implementation check The original #37 defect list is partially stale. Current master already contains `source/runtime-lifecycle.lisp` and `t/runtime-lifecycle-test.lisp`. Already implemented and to be preserved: - a `star.runtime` owner with `:created/:starting/:running/:stopping/:stopped` state; - retained runtime handles for actor system, Rabbit consumers, event consumer, HTTP server, and lparallel kernel; - consumer worker thread references and owned-thread-only joins; - finite Rabbit consume polling so workers can observe stop state; - Clack HTTP handle retention and `clack:stop` shutdown; - Sento actor-system shutdown with `:wait t`; - lparallel kernel shutdown; - startup rollback through `stop-runtime` on startup failure; - SIGTERM/SIGINT stop request handling; - `/live`, `/ready`, `/health` lifecycle endpoints; - tests proving consumer shutdown does not join an unrelated thread and readiness fails closed on Rabbit/CouchDB probe failure. This means ARDR must not propose a second runtime owner. ## Residual correctness gaps ### 1. Runtime ownership is still hard-coded, not closed `star-runtime` has dedicated slots and `start-runtime`/`stop-runtime` directly know each subsystem. New thread/resource-producing components can therefore be added without registering lifecycle ownership, start dependency, stop order, health, or cleanup behavior. Requirement: one closed component/resource registration contract under the existing Common Lisp `star.runtime` authority. Do not create another supervisor/control plane. ### 2. One timeout is repeatedly restarted instead of one shutdown deadline Current stop phases each receive or establish the same timeout independently. Total process shutdown can therefore exceed the operator's intended bound by roughly N phases. Requirement: compute one monotonic absolute shutdown deadline at stop initiation and pass remaining time to every phase/component. Once exhausted, later phases may perform only bounded emergency cleanup/diagnostics; they may not restart the full timeout. ### 3. Cleanup failures are weakly observable Several stop paths use `ignore-errors` or log-and-continue. `stop-runtime` can reach `:stopped` even when a component failed to stop. This loses the distinction between clean shutdown and bounded but incomplete cleanup. Requirement: aggregate typed per-component stop results and terminal shutdown outcome. Preserve cleanup failure evidence while continuing best-effort teardown. ### 4. Reconnect/restart supervision is absent Current Rabbit consumer loop exits on non-timeout transport error; CouchDB readiness is a probe; producer readiness is checked, but there is no runtime-owned reconnect/backoff state machine for Rabbit, CouchDB, or KV dependencies. Requirement: reconnect belongs to the existing runtime/component owner. Use bounded exponential backoff with jitter/cap, cancellation on runtime stop, stable component identity, and readiness degradation while disconnected. Do not spawn detached reconnect threads. ### 5. Readiness is not equivalent to owned-component health Actor/kernel/HTTP readiness is primarily non-NIL handle/state presence. A dead internal worker or failed component can exist behind a retained handle. Requirement: component readiness must be an explicit probe/status from the owning adapter. Runtime readiness is the conjunction of required component states, not object existence. ### 6. No explicit drain contract Stopping HTTP and consumers prevents new intake, but current lifecycle has no general `quiesce -> drain -> stop` component contract, no shared remaining-deadline propagation, and no structured accounting of drained/rejected/handoff work. Requirement: v1 lifecycle phases are `start`, `quiesce`, `drain`, `stop`; components that have no drain work may no-op. #36 owns admission/overload policy; #37 owns when intake closes and how drain deadline propagates. ### 7. Thread-budget accounting must attach to lifecycle ownership #177 requires every runtime-owned thread to map to a component/allocation. Lifecycle registration is the natural owner identity. A component may own zero or more thread allocations/resources, but raw product-side thread creation cannot bypass both registries. ## Evidence-backed library boundary Current upstream Sento exposes actor/context/system shutdown with an optional wait for complete shutdown and dispatcher stop semantics. Keep using that as the actor adapter rather than reimplementing actor teardown. Current Clack exposes server shutdown through the retained handler returned by `clackup`; keep this handle as the HTTP component resource. Bordeaux Threads is the portability layer for thread ownership/joining, not a lifecycle registry. The runtime must therefore retain exact thread/resource ownership rather than enumerate process threads. ## Required lifecycle model Conceptual Common Lisp model, not new StarLang syntax: ```text RuntimeComponent { id dependencies[] required? state resources[] threadAllocationRefs[] start quiesce drain stop readiness reconnectPolicy? } ComponentState = registered | starting | running | degraded | reconnecting | quiescing | draining | stopping | stopped | failed ``` Startup performs a deterministic topological order. Rollback stops only components that successfully crossed into owned started state, in reverse dependency order. Shutdown computes one absolute deadline and executes reverse dependency order through quiesce/drain/stop while accumulating outcomes. ## Reconnect semantics Reconnect applies only to components whose adapter declares it and never runs once runtime state is quiescing/stopping/stopped. Minimum policy: - bounded exponential backoff; - configurable cap and maximum consecutive attempts/window; - jitter to avoid synchronized reconnect storms; - reset after stable healthy period; - readiness false/degraded during reconnect; - no duplicate consumer/producer ownership after reconnect; - reconnect generation/fence so late completion from an older attempt cannot replace the newer resource; - stop cancels pending backoff and fences late reconnect completion. ## StarLang classification No new StarLang syntax is justified. This is host process lifecycle and resource ownership in `starintel-server`. StarLang actors/domain servers may expose lifecycle/health through their existing runtime boundary, but cannot own process thread/transport authority. Implement in Common Lisp using current libraries. No external process or Python controller is justified. ## Adversarial findings Rejected: 1. **Rewrite the existing runtime owner.** Current implementation already supplies the right authority boundary; extend it. 2. **One timeout per stop phase.** Violates the process shutdown bound. 3. **Join `bt:all-threads`.** Can block on or interfere with threads StarIntel does not own. 4. **Mark `:stopped` after swallowed cleanup errors with no result.** Loses operational truth. 5. **Detached reconnect loops.** Create new unowned resources and can resurrect components during shutdown. 6. **Infinite reconnect.** Converts dependency outage into uncontrolled background work. 7. **Readiness = non-NIL handle.** Cannot detect a dead worker behind a retained object. 8. **Force-kill first.** Risks lost Rabbit settlements/outbox/lease state; graceful bounded drain comes first. 9. **Make #37 own overload policy.** #36 owns admission decisions; #37 owns lifecycle transition and deadline propagation. 10. **New StarLang lifecycle grammar.** No language-level semantic gap is demonstrated. ## Mandatory RED-first targets 1. **Absolute deadline fixture:** fake three components whose individual stop calls each consume part of a shared deadline. Untouched current runtime should fail because it restarts timeout windows rather than propagating one absolute deadline. 2. **Partial rollback fixture:** component B start fails after A starts; prove only successfully started components are stopped exactly once in reverse dependency order and cleanup failures remain in the returned shutdown report. 3. **Reconnect fencing fixture:** reconnect attempt generation N completes after N+1 or after stop; stale completion must be rejected and may not install a resource. 4. **Readiness fixture:** retained component handle with failed explicit health status must make runtime unready. 5. **Ownership fixture:** a runtime-owned thread/resource without a registered component owner must fail lifecycle/accounting validation. ## Decision Evidence is sufficient to proceed directly to design. The architecture boundary is clear and current implementation provides the seed rather than requiring a rewrite. Implementation approval: `PENDING / AWAITING_OPERATOR_APPROVAL`.