nsaspy/starintel-auto-research

Fork 0

DESIGN_READY_FOR_OPERATOR_REVIEW: StarLang bounded OSINT orchestration kernel #168

New issue

Open

opened 2026-08-29 10:28:06 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-08-29 10:28:06 +00:00

Owner

Authority / state

Source research: #165
Related Source Acquisition design: #171
Related expert-system direction: #156 / approved StarLang expert research
ARDR policy: #170
Research state: READY_FOR_DESIGN
Design state: DESIGN_READY_FOR_OPERATOR_REVIEW
Implementation approval: PENDING — operator only

This design does not authorize implementation and must not enter the executable RAGE queue until the operator explicitly approves a slice.

Problem

StarIntel OSINT domains need reusable bounded fan-out, pivot/traversal, step binding, conditional outcome handling and partial aggregation. Today those behaviors risk being reimplemented independently by Identity, Infrastructure, Source Acquisition, agentic researchers and external-tool wrappers.

The fix is deliberately smaller than a generic workflow engine: add only the closed orchestration semantics that belong in StarLang, and keep provider health, scheduling, correlation, transport policy, dedupe and change detection in their proper runtime/domain owners.

Architecture boundary

StarLang owns

A closed typed orchestration IR for:

bounded fan-out over an already-authorized capability/provider set;
bounded traversal/pivots with visited identity and cycle control;
typed step/result bindings;
branch-on-typed-outcome;
bounded aggregation with explicit partial semantics;
causal/correlation linkage for cross-domain invocation;
explicit budgets: max branches, depth, items, bytes, operations and deadline;
deterministic orchestration definition/version identity.

Runtime/domain libraries own

provider registry discovery and versioned registry refresh;
provider health, cooldown, circuit breaker and demotion;
quota/rate metadata;
scheduled/stream trigger actors;
transport/proxy/Tor/I2P adapters;
provenance-aware dedupe and change-set generation;
persistent checkpoints/visited-state storage using existing journal/artifact ownership.

Expert system owns

Correlation rules, explanations and proof traces. A playbook may request an expert decision or consume an expert conclusion, but it does not embed a second correlation-rule DSL.

Source Acquisition reuse

Reuse #171 for bounded traversal/request fingerprints, opaque credential/session/artifact/result refs, challenge continuation and source-provenance semantics. Do not duplicate those primitives.

Proposed closed IR

Conceptual only; final syntax follows StarLang conventions.

(playbook enumerate-identity
  (:input identity-seed
   :emits account-observation
   :budget (:branches 32 :depth 2 :items 500 :deadline-ms 30000))

  (fan-out provider from (capabilities identity.account-probe)
    (:max 32 :concurrency 8)
    (call provider probe seed -> result))

  (collect result
    (:outcomes (success partial blocked failed))
    -> observations)

  (when-outcome observations partial
    (emit-warning partial-provider-coverage))

  (pivot observation
    (:max-depth 2 :visited account-id)
    ...))

No plan step may name arbitrary Common Lisp functions, shell commands, provider secrets, network endpoints outside typed capability refs, or unbounded recursion/fan-out.

Outcome model

Generic orchestration results must distinguish at least:

success
partial
blocked
unavailable
cancelled
failed

Aggregation retains per-branch/provider outcomes and provenance. partial is not silently coerced to success or absence.

Provider registry / health contract

A reusable runtime component exposes authorized provider capability entries containing stable provider ID/version/capabilities plus scoped health metadata.

Health state should include bounded failure counters, last success/failure class, cooldown/open-until state, and optional rate/quota observations. Health is policy input, not truth about the source itself.

Important: health/demotion state may need principal/config scope because different credentials, quotas or egress paths can produce different provider behavior.

Dynamic activation

Newly emitted typed documents may request matching capabilities through a registry lookup, but emitted data cannot create executable capabilities or arbitrary handlers. Activation is selection from pre-registered, authorized capability definitions only.

Scheduling / streams

Do not add cron or a stream engine to StarLang syntax. Scheduler/stream actors send typed trigger messages into ordinary domain/playbook entrypoints. Trigger state and checkpointing remain runtime/domain state.

Persistence / restart

orchestration definition/version is durable data;
in-flight checkpoint state reuses StarLang journal/artifact ownership;
visited sets and branch outcomes are bounded and checkpointable;
restart may resume only from a valid checkpoint tied to the same orchestration definition and input identity;
effectful operations require existing idempotency/fencing semantics and must never be replayed merely because a pure branch is retryable.

Adversarial review

Rejected

Generic workflow engine inside StarLang — too broad; creates a second control plane.
Provider health/circuit breaker syntax — runtime policy, not language semantics.
Correlation rules in playbooks — duplicates StarLang expert/logic ownership.
Global provider-health booleans — can mix tenant/credential/egress-specific failures.
Unbounded recursive pivots — denial-of-service and crawl explosion risk.
Dynamic execution from emitted document content — capability escalation risk.
Automatic replay of effectful branches — duplicates side effects and violates idempotency.

Required implementation invariants

every fan-out/traversal form has static positive ceilings;
runtime enforces total aggregate budget, not merely per-branch limits;
cancellation fences late branch completions;
branch results preserve provider/cause/provenance identity;
provider registry cannot grant capabilities absent caller authorization;
partial aggregation remains visible to downstream expert/research logic;
restart cannot silently switch provider registry version without recording the change.

Dependency-ordered proposed slices

All are AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL.

StarLang orchestration IR — typed playbook, bounded fan-out, bindings, branch-on-outcome, aggregate result contract.
Bounded pivot/traversal integration — reuse #171 visited/cycle/budget semantics rather than duplicate them.
Causal cross-domain invocation — preserve correlation/cause/provenance across local and remote calls.
Provider registry + scoped health library — runtime/domain component, not compiler syntax.
Scheduler/stream trigger actor pattern — typed triggers into existing playbooks/domain servers.
OSINT integration fixtures — Identity account enumeration and Infrastructure discovery as proof cases.

RED-first targets

Slice 1

A valid StarLang fixture declaring a bounded playbook with fan-out, result binding and partial aggregation must fail on untouched current StarLang because no playbook/fan-out orchestration declaration exists.

Negative fixtures must prove rejection of:

fan-out without a positive maximum;
recursion/pivot without depth + visited identity;
arbitrary host function invocation;
raw credential literals;
unknown outcome branch;
aggregation without a total budget.

Slice 4

A deterministic fake-provider test must show a failing provider transitions through bounded health/cooldown policy without changing authorization or globally demoting an independent principal/provider context.

Acceptance

Identity and Infrastructure domains can express bounded multi-provider execution without custom concurrency controllers;
recursive pivots are cycle- and budget-bounded;
provider partial failures remain typed and provenance-preserving;
expert correlation remains in the expert engine;
scheduled ingestion remains actor-driven;
local and remote cross-domain calls preserve the same cause/provenance contract;
no second workflow/control plane is introduced;
implementation remains blocked until explicit operator approval.

Implementation approval

PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL

ARDR has no authority to change this state.

## Authority / state - Source research: #165 - Related Source Acquisition design: #171 - Related expert-system direction: #156 / approved StarLang expert research - ARDR policy: #170 - Research state: `READY_FOR_DESIGN` - Design state: `DESIGN_READY_FOR_OPERATOR_REVIEW` - **Implementation approval: PENDING — operator only** This design does not authorize implementation and must not enter the executable RAGE queue until the operator explicitly approves a slice. ## Problem StarIntel OSINT domains need reusable bounded fan-out, pivot/traversal, step binding, conditional outcome handling and partial aggregation. Today those behaviors risk being reimplemented independently by Identity, Infrastructure, Source Acquisition, agentic researchers and external-tool wrappers. The fix is deliberately **smaller than a generic workflow engine**: add only the closed orchestration semantics that belong in StarLang, and keep provider health, scheduling, correlation, transport policy, dedupe and change detection in their proper runtime/domain owners. ## Architecture boundary ### StarLang owns A closed typed orchestration IR for: - bounded fan-out over an already-authorized capability/provider set; - bounded traversal/pivots with visited identity and cycle control; - typed step/result bindings; - branch-on-typed-outcome; - bounded aggregation with explicit partial semantics; - causal/correlation linkage for cross-domain invocation; - explicit budgets: max branches, depth, items, bytes, operations and deadline; - deterministic orchestration definition/version identity. ### Runtime/domain libraries own - provider registry discovery and versioned registry refresh; - provider health, cooldown, circuit breaker and demotion; - quota/rate metadata; - scheduled/stream trigger actors; - transport/proxy/Tor/I2P adapters; - provenance-aware dedupe and change-set generation; - persistent checkpoints/visited-state storage using existing journal/artifact ownership. ### Expert system owns Correlation rules, explanations and proof traces. A playbook may request an expert decision or consume an expert conclusion, but it does not embed a second correlation-rule DSL. ### Source Acquisition reuse Reuse #171 for bounded traversal/request fingerprints, opaque credential/session/artifact/result refs, challenge continuation and source-provenance semantics. Do not duplicate those primitives. ## Proposed closed IR Conceptual only; final syntax follows StarLang conventions. ```lisp (playbook enumerate-identity (:input identity-seed :emits account-observation :budget (:branches 32 :depth 2 :items 500 :deadline-ms 30000)) (fan-out provider from (capabilities identity.account-probe) (:max 32 :concurrency 8) (call provider probe seed -> result)) (collect result (:outcomes (success partial blocked failed)) -> observations) (when-outcome observations partial (emit-warning partial-provider-coverage)) (pivot observation (:max-depth 2 :visited account-id) ...)) ``` No plan step may name arbitrary Common Lisp functions, shell commands, provider secrets, network endpoints outside typed capability refs, or unbounded recursion/fan-out. ## Outcome model Generic orchestration results must distinguish at least: - `success` - `partial` - `blocked` - `unavailable` - `cancelled` - `failed` Aggregation retains per-branch/provider outcomes and provenance. `partial` is not silently coerced to success or absence. ## Provider registry / health contract A reusable runtime component exposes authorized provider capability entries containing stable provider ID/version/capabilities plus scoped health metadata. Health state should include bounded failure counters, last success/failure class, cooldown/open-until state, and optional rate/quota observations. Health is policy input, not truth about the source itself. Important: health/demotion state may need principal/config scope because different credentials, quotas or egress paths can produce different provider behavior. ## Dynamic activation Newly emitted typed documents may request matching capabilities through a registry lookup, but emitted data cannot create executable capabilities or arbitrary handlers. Activation is selection from pre-registered, authorized capability definitions only. ## Scheduling / streams Do not add cron or a stream engine to StarLang syntax. Scheduler/stream actors send typed trigger messages into ordinary domain/playbook entrypoints. Trigger state and checkpointing remain runtime/domain state. ## Persistence / restart - orchestration definition/version is durable data; - in-flight checkpoint state reuses StarLang journal/artifact ownership; - visited sets and branch outcomes are bounded and checkpointable; - restart may resume only from a valid checkpoint tied to the same orchestration definition and input identity; - effectful operations require existing idempotency/fencing semantics and must never be replayed merely because a pure branch is retryable. ## Adversarial review ### Rejected 1. **Generic workflow engine inside StarLang** — too broad; creates a second control plane. 2. **Provider health/circuit breaker syntax** — runtime policy, not language semantics. 3. **Correlation rules in playbooks** — duplicates StarLang expert/logic ownership. 4. **Global provider-health booleans** — can mix tenant/credential/egress-specific failures. 5. **Unbounded recursive pivots** — denial-of-service and crawl explosion risk. 6. **Dynamic execution from emitted document content** — capability escalation risk. 7. **Automatic replay of effectful branches** — duplicates side effects and violates idempotency. ### Required implementation invariants - every fan-out/traversal form has static positive ceilings; - runtime enforces total aggregate budget, not merely per-branch limits; - cancellation fences late branch completions; - branch results preserve provider/cause/provenance identity; - provider registry cannot grant capabilities absent caller authorization; - partial aggregation remains visible to downstream expert/research logic; - restart cannot silently switch provider registry version without recording the change. ## Dependency-ordered proposed slices All are **AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL**. 1. **StarLang orchestration IR** — typed `playbook`, bounded `fan-out`, bindings, branch-on-outcome, aggregate result contract. 2. **Bounded pivot/traversal integration** — reuse #171 visited/cycle/budget semantics rather than duplicate them. 3. **Causal cross-domain invocation** — preserve correlation/cause/provenance across local and remote calls. 4. **Provider registry + scoped health library** — runtime/domain component, not compiler syntax. 5. **Scheduler/stream trigger actor pattern** — typed triggers into existing playbooks/domain servers. 6. **OSINT integration fixtures** — Identity account enumeration and Infrastructure discovery as proof cases. ## RED-first targets ### Slice 1 A valid StarLang fixture declaring a bounded playbook with fan-out, result binding and partial aggregation must fail on untouched current StarLang because no playbook/fan-out orchestration declaration exists. Negative fixtures must prove rejection of: - fan-out without a positive maximum; - recursion/pivot without depth + visited identity; - arbitrary host function invocation; - raw credential literals; - unknown outcome branch; - aggregation without a total budget. ### Slice 4 A deterministic fake-provider test must show a failing provider transitions through bounded health/cooldown policy without changing authorization or globally demoting an independent principal/provider context. ## Acceptance - Identity and Infrastructure domains can express bounded multi-provider execution without custom concurrency controllers; - recursive pivots are cycle- and budget-bounded; - provider partial failures remain typed and provenance-preserving; - expert correlation remains in the expert engine; - scheduled ingestion remains actor-driven; - local and remote cross-domain calls preserve the same cause/provenance contract; - no second workflow/control plane is introduced; - implementation remains blocked until explicit operator approval. ## Implementation approval `PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL` ARDR has no authority to change this state.

nsaspy referenced this issue

2026-08-29 10:28:06 +00:00

DESIGN_READY_FOR_OPERATOR_REVIEW: StarLang Source Acquisition / Scraping Domain Server #169

nsaspy referenced this issue

2026-08-29 10:28:06 +00:00

Research + design: StarLang-defined scraping domain actor and researcher integration #171