StarLang research: OSINT domain-server orchestration gaps #175

New issue

Open

opened 2026-08-29 10:28:07 +00:00 by nsaspy · 1 comment

nsaspy commented

2026-08-29 10:28:07 +00:00

Owner

Goal

Map the OSINT feature research against the current StarLang implementation before proposing any Common Lisp or Python fallback.

Verified existing StarLang/runtime primitives

Current lost-rob0t/star-lang already has substantial support relevant to OSINT:

domain-server compilation with owned document types, accepted messages, declared tools, dispatcher, restart, mailbox, and capabilities
typed domain tools with input/output message types and positive timeout-ms
process-tool runner with hard timeout / TERM / kill-after behavior
deterministic wire dispatcher
command deadlines
cancellation by message/correlation
explicit retry outcomes and retry-after metadata
retryable vs terminal failures
idempotency identity and replay of terminal outcomes
runtime journal append/replay with pending / route-result / remote-result events
validated journal ordering

Relevant implementation locations include:

prototype/domain-server-core-prototype.lisp
starlang-runtime/src/wire-dispatcher.lisp
star-journal/src/journal.lisp

OSINT semantics still requiring verification/design

Do not assume these are missing until each is checked against the current compiler/runtime:

bounded parallel fan-out over a dynamic provider registry
bounded recursive pivots with visited-set/cycle control
declarative playbook composition and result bindings
conditional/failure branches in playbooks
dynamic activation of capabilities from newly emitted document types
persistent provider health/drift state
circuit breaker / provider demotion semantics
registry refresh/version pinning
typed partial-result aggregation across many providers
provenance-preserving deduplication and multi-source confirmation
correlation-rule declarations with explanations
cross-domain invocation preserving cause/provenance
scan profiles/budgets such as fast/balanced/exhaustive
proxy/Tor/I2P transport policy as an adapter selection concern
authenticated session references and expiry without embedding raw credentials in plans
scheduled/stream ingestor triggers
change-set generation between repeated runs

Required output

For every semantic above classify exactly one:

already supported — point to compiler/runtime contract and tests
expressible now — no language change, only a domain library/pattern
StarLang extension needed — define syntax/IR/runtime/test requirement
Common Lisp adapter needed — only when it is an implementation primitive below StarLang
external process required — only when neither StarLang nor Common Lisp is a reasonable implementation boundary

Python is not an automatic fallback. External Python is permitted only as the last case and must stay behind star-process-port / domain-tool style boundaries.

#159 MailAccess
#160 username/account enumeration
#161 email/Google identity
#162 infrastructure discovery
#163 SpiderFoot orchestration/correlation
#164 IntelOwl analysis lifecycle

Output gate

Keep this as research staging. Do not promote conclusions into ARARD/ARADR documents until explicit human instruction to add research.

## Goal Map the OSINT feature research against the **current** StarLang implementation before proposing any Common Lisp or Python fallback. ## Verified existing StarLang/runtime primitives Current `lost-rob0t/star-lang` already has substantial support relevant to OSINT: - `domain-server` compilation with owned document types, accepted messages, declared tools, dispatcher, restart, mailbox, and capabilities - typed domain tools with input/output message types and positive `timeout-ms` - process-tool runner with hard timeout / TERM / kill-after behavior - deterministic wire dispatcher - command deadlines - cancellation by message/correlation - explicit retry outcomes and retry-after metadata - retryable vs terminal failures - idempotency identity and replay of terminal outcomes - runtime journal append/replay with pending / route-result / remote-result events - validated journal ordering Relevant implementation locations include: - `prototype/domain-server-core-prototype.lisp` - `starlang-runtime/src/wire-dispatcher.lisp` - `star-journal/src/journal.lisp` ## OSINT semantics still requiring verification/design Do not assume these are missing until each is checked against the current compiler/runtime: - bounded parallel fan-out over a dynamic provider registry - bounded recursive pivots with visited-set/cycle control - declarative playbook composition and result bindings - conditional/failure branches in playbooks - dynamic activation of capabilities from newly emitted document types - persistent provider health/drift state - circuit breaker / provider demotion semantics - registry refresh/version pinning - typed partial-result aggregation across many providers - provenance-preserving deduplication and multi-source confirmation - correlation-rule declarations with explanations - cross-domain invocation preserving cause/provenance - scan profiles/budgets such as fast/balanced/exhaustive - proxy/Tor/I2P transport policy as an adapter selection concern - authenticated session references and expiry without embedding raw credentials in plans - scheduled/stream ingestor triggers - change-set generation between repeated runs ## Required output For every semantic above classify exactly one: 1. **already supported** — point to compiler/runtime contract and tests 2. **expressible now** — no language change, only a domain library/pattern 3. **StarLang extension needed** — define syntax/IR/runtime/test requirement 4. **Common Lisp adapter needed** — only when it is an implementation primitive below StarLang 5. **external process required** — only when neither StarLang nor Common Lisp is a reasonable implementation boundary Python is not an automatic fallback. External Python is permitted only as the last case and must stay behind `star-process-port` / domain-tool style boundaries. ## Related OSINT research issues - #159 MailAccess - #160 username/account enumeration - #161 email/Google identity - #162 infrastructure discovery - #163 SpiderFoot orchestration/correlation - #164 IntelOwl analysis lifecycle ## Output gate Keep this as research staging. Do not promote conclusions into ARARD/ARADR documents until explicit human instruction to add research.

nsaspy commented

2026-08-29 10:28:17 +00:00

Author

Owner

ARDR checkpoint — current StarLang classification

Selected as an unlabelled dependency-unblocking research item because #159–#164 and the Source Acquisition design (#171) depend on knowing which orchestration semantics belong in StarLang versus reusable domain/runtime libraries.

Current implementation inspected

Current lost-rob0t/star-lang remains Common-Lisp-only and explicitly owns actor protocol, mailbox, supervision, journal, lease, capability, artifact, adapter SDK, HTTP/process ports, compiler and durable runtime systems. The authoritative working implementation remains under prototype/; the README explicitly says current conformance is still incomplete, so this pass does not infer implementation merely from approved research.

prototype/domain-server-core-prototype.lisp currently proves domain-server compilation with typed owned documents/accepted messages/tools, restart/mailbox/dispatcher/capability declarations, keyed runtime instances, injected tool-runner ports, and hard process-tool timeouts. Existing #165 evidence plus current remoting/journal prototypes already establish deadlines, cancellation, retry outcomes, idempotency/terminal replay, remote-result journaling and local/remote domain-server execution.

Classification

For the unresolved semantics in this issue:

Semantic	Classification	StarIntel consequence
bounded parallel fan-out over dynamic provider registry	3 — StarLang extension needed for closed bounded fan-out/aggregation IR; provider discovery itself is runtime/library	avoid every Identity/Infra actor writing ad-hoc concurrency loops
bounded recursive pivots + visited/cycle control	3 — StarLang extension needed, shared with #171 bounded traversal	one closed traversal primitive, not SpiderFoot-style hidden recursion
declarative playbook composition + result bindings	3 — StarLang extension needed	small closed orchestration DAG/step IR; not a second programming language
conditional/failure branches in playbooks	3 — StarLang extension needed	typed branch-on-outcome forms over closed outcomes
dynamic activation from emitted document types	2 — expressible as runtime/library pattern using type/capability registry + supervisor/dispatcher; no syntax required initially	keep activation policy outside document schemas
persistent provider health/drift state	2 — runtime/domain library	provider registry actor/domain server owns state, not compiler
circuit breaker/provider demotion	2 — runtime/domain library	reusable provider-health policy beneath source/domain actors
registry refresh/version pinning	2 — runtime/domain library	versioned registry artifacts/config + explicit refresh actor
typed partial-result aggregation	3 — StarLang extension needed for generic bounded aggregation/outcome IR	one partial/success/blocked/failure aggregation contract
provenance-preserving dedupe / multi-source confirmation	2 — domain/ontology library, coordinated with STAR-RESEARCH-012	do not make dedupe a language primitive
correlation-rule declarations + explanations	2 — StarLang expert/logic surface already the architectural owner	do not add a second correlation DSL inside playbooks
cross-domain invocation preserving cause/provenance	3 — small lifecycle/runtime extension: invocation already exists; causal/provenance linkage must be explicit and portable	same semantics local/remote
fast/balanced/exhaustive scan profiles/budgets	2 — library/config policy over explicit budgets	profiles compile to bounded parameters; no magic runtime mode
proxy/Tor/I2P policy	4 — Common Lisp adapter/runtime concern selected through typed transport capability	transport stays beneath StarLang definitions
authenticated session refs + expiry	3 — StarLang/runtime opaque-ref extension, already converging with #171	raw credentials never enter plans/source
scheduled/stream ingestor triggers	2 — scheduler/stream actor pattern; only typed trigger messages are needed	no cron/event engine in compiler
change-set generation between repeated runs	2 — domain library over versioned observations/provenance	keep diff semantics near document/observation model

Important negative findings

Do not add circuit breakers, provider health, scheduling, Tor policy, dedupe, or change detection as syntax. They are reusable runtime/domain services.
Do not create a generic workflow language parallel to StarLang. The missing language surface is only closed bounded orchestration: fan-out, traversal, step bindings, typed branches, aggregation and causal invocation.
Do not make correlation a playbook feature. StarLang expert/logic semantics own rule firing and explanations.
Do not let dynamic capability activation execute arbitrary code from emitted data. Emitted document type may select from an already-authorized capability registry only.
Provider health/demotion state must be principal/config scoped where credentials/quotas differ; a global health boolean can leak one tenant's failure state into another.

Research decision

Evidence is sufficient to mark #165 READY_FOR_DESIGN under ARDR policy. The design should split:

minimal StarLang bounded-orchestration IR;
provider-registry/health runtime library;
expert-system correlation reuse;
Source Acquisition traversal/opaque-ref reuse from #171;
scheduler/stream triggers as actors rather than compiler syntax.

Implementation remains operator-only. No implementation approval is inferred from this checkpoint.

## ARDR checkpoint — current StarLang classification Selected as an unlabelled **dependency-unblocking** research item because #159–#164 and the Source Acquisition design (#171) depend on knowing which orchestration semantics belong in StarLang versus reusable domain/runtime libraries. ### Current implementation inspected Current `lost-rob0t/star-lang` remains Common-Lisp-only and explicitly owns actor protocol, mailbox, supervision, journal, lease, capability, artifact, adapter SDK, HTTP/process ports, compiler and durable runtime systems. The authoritative working implementation remains under `prototype/`; the README explicitly says current conformance is still incomplete, so this pass does not infer implementation merely from approved research. `prototype/domain-server-core-prototype.lisp` currently proves domain-server compilation with typed owned documents/accepted messages/tools, restart/mailbox/dispatcher/capability declarations, keyed runtime instances, injected tool-runner ports, and hard process-tool timeouts. Existing #165 evidence plus current remoting/journal prototypes already establish deadlines, cancellation, retry outcomes, idempotency/terminal replay, remote-result journaling and local/remote domain-server execution. ### Classification For the unresolved semantics in this issue: | Semantic | Classification | StarIntel consequence | |---|---|---| | bounded parallel fan-out over dynamic provider registry | **3 — StarLang extension needed** for closed bounded fan-out/aggregation IR; provider discovery itself is runtime/library | avoid every Identity/Infra actor writing ad-hoc concurrency loops | | bounded recursive pivots + visited/cycle control | **3 — StarLang extension needed**, shared with #171 bounded traversal | one closed traversal primitive, not SpiderFoot-style hidden recursion | | declarative playbook composition + result bindings | **3 — StarLang extension needed** | small closed orchestration DAG/step IR; not a second programming language | | conditional/failure branches in playbooks | **3 — StarLang extension needed** | typed branch-on-outcome forms over closed outcomes | | dynamic activation from emitted document types | **2 — expressible as runtime/library pattern** using type/capability registry + supervisor/dispatcher; no syntax required initially | keep activation policy outside document schemas | | persistent provider health/drift state | **2 — runtime/domain library** | provider registry actor/domain server owns state, not compiler | | circuit breaker/provider demotion | **2 — runtime/domain library** | reusable provider-health policy beneath source/domain actors | | registry refresh/version pinning | **2 — runtime/domain library** | versioned registry artifacts/config + explicit refresh actor | | typed partial-result aggregation | **3 — StarLang extension needed** for generic bounded aggregation/outcome IR | one partial/success/blocked/failure aggregation contract | | provenance-preserving dedupe / multi-source confirmation | **2 — domain/ontology library**, coordinated with STAR-RESEARCH-012 | do not make dedupe a language primitive | | correlation-rule declarations + explanations | **2 — StarLang expert/logic surface** already the architectural owner | do not add a second correlation DSL inside playbooks | | cross-domain invocation preserving cause/provenance | **3 — small lifecycle/runtime extension**: invocation already exists; causal/provenance linkage must be explicit and portable | same semantics local/remote | | fast/balanced/exhaustive scan profiles/budgets | **2 — library/config policy** over explicit budgets | profiles compile to bounded parameters; no magic runtime mode | | proxy/Tor/I2P policy | **4 — Common Lisp adapter/runtime concern** selected through typed transport capability | transport stays beneath StarLang definitions | | authenticated session refs + expiry | **3 — StarLang/runtime opaque-ref extension**, already converging with #171 | raw credentials never enter plans/source | | scheduled/stream ingestor triggers | **2 — scheduler/stream actor pattern**; only typed trigger messages are needed | no cron/event engine in compiler | | change-set generation between repeated runs | **2 — domain library** over versioned observations/provenance | keep diff semantics near document/observation model | ### Important negative findings - **Do not add circuit breakers, provider health, scheduling, Tor policy, dedupe, or change detection as syntax.** They are reusable runtime/domain services. - **Do not create a generic workflow language parallel to StarLang.** The missing language surface is only closed bounded orchestration: fan-out, traversal, step bindings, typed branches, aggregation and causal invocation. - **Do not make correlation a playbook feature.** StarLang expert/logic semantics own rule firing and explanations. - **Do not let dynamic capability activation execute arbitrary code from emitted data.** Emitted document type may select from an already-authorized capability registry only. - Provider health/demotion state must be principal/config scoped where credentials/quotas differ; a global health boolean can leak one tenant's failure state into another. ### Research decision Evidence is sufficient to mark #165 **READY_FOR_DESIGN** under ARDR policy. The design should split: 1. minimal StarLang bounded-orchestration IR; 2. provider-registry/health runtime library; 3. expert-system correlation reuse; 4. Source Acquisition traversal/opaque-ref reuse from #171; 5. scheduler/stream triggers as actors rather than compiler syntax. Implementation remains operator-only. No implementation approval is inferred from this checkpoint.