nsaspy/star-lang

Fork 0

Epic: make StarLang safely embeddable in StarIntel services #57

New issue

Open

opened 2026-09-02 17:26:05 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-02 17:26:05 +00:00

Owner

Goal

Make StarLang a stable embeddable Common Lisp library/runtime that starintel-server and other StarIntel services can load in-process without depending on prototype/, private implementation packages, CLI entrypoints, global process state, or ad-hoc host callbacks.

The first concrete downstream consumer is starintel-server document matching/transforms from server issue #44 / STAR-RESEARCH-051. Matcher definitions and matcher evaluation semantics are moving into StarLang. The server should embed StarLang and provide typed host/application ports; it should not grow a second matcher DSL or retain a Common Lisp callback registry.

Related StarLang gates:

#47 — productionize StarLang and retire prototype authority
#52 — prove final-system port complete, then delete prototype/
#6 — research 000–009 conformance

Downstream seed:

https://github.com/lost-rob0t/starintel-server/issues/44

Current readiness determination

NOT READY YET as a stable production embedding boundary.

There is substantial usable implementation already:

final starlang-runtime exposes concrete actor/runtime/dispatcher APIs;
the prototype public API exposes parsing, expansion, validation, compilation, loading, document construction, encoding, decoding, relations, and ID generation;
ASDF/Nix infrastructure already exists.

But the production embedding boundary is not complete:

the complete parser/compiler/public load API is still prototype-owned;
prototype/star-lang-api.lisp delegates to star-lang.core-surface.prototype, star-lang.loader, constructor-runtime, and document-runtime prototype-era ownership;
final starlang-compiler does not yet own the complete closed parser → expand → validate → normalized IR pipeline;
#47 and #52 explicitly track retirement of prototype authority;
#6 still marks research conformance incomplete.

This epic is therefore the consumer-facing readiness gate for embedding. It does not replace #47/#52; it defines the stable API and integration proof those efforts must expose to downstream services.

Required public embedding surface

Create final-system, documented APIs with no prototype package dependency for at least:

Compiler / program lifecycle

parse StarLang from bounded UTF-8 bytes/string/path through the closed parser;
resolve only explicitly configured locked imports/effect ports;
expand, validate, and compile to deterministic runtime-neutral IR;
return structured diagnostics/conditions without entering the debugger;
expose compiler/build/schema identity and compiled-program digest;
support compile-only/check-only operation without starting a runtime.

Runtime lifecycle

create an isolated runtime instance from explicit configuration;
instantiate/load a compiled StarLang program into that runtime;
start/stop/shutdown deterministically;
no implicit singleton runtime or required global registry;
multiple independent StarLang runtimes must coexist in one SBCL image;
all background threads/workers are runtime-owned and joinable on shutdown;
failures are structured conditions/results, not process exits.

Host ports / capabilities

hosts provide explicitly typed ports/capabilities to StarLang;
no arbitrary Common Lisp function object may be embedded in .star source or normalized IR;
no host package names, actor refs, broker handles, database handles, credentials, or Lisp objects may leak into portable IR;
effect/adapter lookup is explicit and instance-scoped;
host callbacks, where an adapter necessarily needs one internally, remain behind a closed final-system port contract and are not StarLang language values.

Matcher / document-transform support

Add a StarLang-owned bounded matcher/transform semantic surface sufficient for STAR-RESEARCH-051:

typed accepted input document/schema set;
typed emitted document/schema set;
field selectors over canonical documents;
deterministic predicates/combinators;
bounded textual matching/extraction;
closed URI candidate/parse/normalize primitive suitable for HTTP(S) URL extraction;
per-input and aggregate output ceilings;
deadline/budget propagation;
typed transform failure/result;
provenance metadata describing source document, transform identity/version, selected field/location, and run/correlation identity.

The matcher language must be StarLang. Do not expose PCRE strings plus Lisp callbacks as the canonical model. Native Common Lisp implementation details may exist inside the StarLang runtime for closed primitives, but downstream hosts invoke StarLang semantics, not those implementation functions directly.

Downstream integration contract

starintel-server must be able to:

ASDF-load only final StarLang systems;
compile/load approved .star matcher/transform definitions;
bind typed application ports such as canonical document ingest;
run matcher/transform programs in an explicit StarLang runtime instance;
receive typed generated-document results/provenance;
submit those results through the server's canonical ingest boundary;
replace/restart the StarLang runtime without stale Lisp/actor references surviving the generation boundary;
shut the server down with no leaked StarLang threads/resources.

The server remains authoritative for persistence, authentication/authorization, deployment policy, runtime resource admission, and canonical ingest. StarLang remains authoritative for language semantics and matcher/transform evaluation.

Required isolation and safety properties

bounded source bytes, parser depth/nodes/tokens, compilation work, transform input bytes, candidates, outputs, and runtime deadlines;
deterministic behavior for pure matcher/transform programs;
no Common Lisp READ/EVAL of .star source;
no ambient credentials or filesystem/network access;
outbound effects only through explicit capabilities/ports;
no process-global mutable matcher registry;
no raw actor refs retained by compiled programs;
runtime shutdown/restart invalidates runtime-local handles cleanly;
embedding failures cannot call quit, terminate SBCL, or enter an interactive debugger.

Implementation slices

Slice 1 — embedding API contract and RED tests

define final public packages/system dependencies;
specify runtime/compiler/program handles and structured conditions;
add fresh-process tests that load only final systems;
add two-runtime isolation tests;
add shutdown/no-leak tests;
add static guards against prototype package dependencies.

Slice 2 — final compiler embedding path

move/finish closed parser, expansion, validation, normalized IR, loader and import/effect boundaries under final systems;
expose compile/check APIs;
satisfy the relevant #6/#47/#52 gates.

Slice 3 — final runtime embedding path

expose create/load/start/stop/shutdown APIs around final starlang-runtime;
make runtime-owned resources instance-scoped;
prove multiple runtimes coexist safely.

Slice 4 — StarLang matcher/transform semantics

implement typed matcher/transform declarations and IR;
implement bounded selectors/predicates/extraction primitives;
add HTTP(S) URI extraction fixtures;
prohibit arbitrary host callbacks/raw Lisp objects in declarations and IR.

Slice 5 — StarIntel integration proof

Using a test consumer or starintel-server integration branch:

load final StarLang systems only;
compile a .star URL matcher/transform;
feed a canonical test document;
obtain typed URL + relation output with provenance;
route output into a fake/contract canonical-ingest port;
restart runtime and prove stale references fail;
shutdown and prove no threads/resources leak.

Acceptance gate: `STARLANG_EMBED_READY`

Do not claim StarLang production-embedding readiness until all are true:

starlang-compiler owns the complete production compiler path.
public embedding API imports no *.prototype package.
normal embedding path does not load starlang-prototype or files under prototype/.
#6 conformance blockers relevant to parsing/compiler/wire contracts are closed.
#47/#52 final-system migration gates required by the embedding path are green.
two isolated runtimes can coexist in one process.
deterministic shutdown leaves no StarLang-owned thread/process/resource behind.
compiled programs contain no raw Common Lisp callbacks/objects or raw actor refs.
bounded StarLang matcher/transform semantics exist and pass URL extraction fixtures.
a downstream StarIntel integration test compiles and runs a .star transform through final systems only.
Nix + ASDF + CI run the same embedding test matrix from a clean environment.

Non-goals

StarLang does not become StarIntel persistence authority.
StarLang does not own server authentication/authorization policy.
Do not add a second out-of-process language server just to embed StarLang; in-process Common Lisp embedding is the target.
Do not preserve the old server matcher callback registry as an alternate authority.
Do not declare readiness merely because prototype/star-lang-api.lisp can be called today.

Definition of done

This epic closes when STARLANG_EMBED_READY is green and a StarIntel consumer can depend only on released/final StarLang systems to compile and execute bounded StarLang programs in-process, including the matcher/transform vertical required by server #44.

## Goal Make StarLang a **stable embeddable Common Lisp library/runtime** that `starintel-server` and other StarIntel services can load in-process without depending on `prototype/`, private implementation packages, CLI entrypoints, global process state, or ad-hoc host callbacks. The first concrete downstream consumer is `starintel-server` document matching/transforms from server issue #44 / `STAR-RESEARCH-051`. Matcher definitions and matcher evaluation semantics are moving into StarLang. The server should embed StarLang and provide typed host/application ports; it should not grow a second matcher DSL or retain a Common Lisp callback registry. Related StarLang gates: - #47 — productionize StarLang and retire prototype authority - #52 — prove final-system port complete, then delete `prototype/` - #6 — research 000–009 conformance Downstream seed: - https://github.com/lost-rob0t/starintel-server/issues/44 ## Current readiness determination **NOT READY YET as a stable production embedding boundary.** There is substantial usable implementation already: - final `starlang-runtime` exposes concrete actor/runtime/dispatcher APIs; - the prototype public API exposes parsing, expansion, validation, compilation, loading, document construction, encoding, decoding, relations, and ID generation; - ASDF/Nix infrastructure already exists. But the production embedding boundary is not complete: - the complete parser/compiler/public load API is still prototype-owned; - `prototype/star-lang-api.lisp` delegates to `star-lang.core-surface.prototype`, `star-lang.loader`, constructor-runtime, and document-runtime prototype-era ownership; - final `starlang-compiler` does not yet own the complete closed parser → expand → validate → normalized IR pipeline; - #47 and #52 explicitly track retirement of prototype authority; - #6 still marks research conformance incomplete. This epic is therefore the **consumer-facing readiness gate** for embedding. It does not replace #47/#52; it defines the stable API and integration proof those efforts must expose to downstream services. ## Required public embedding surface Create final-system, documented APIs with no prototype package dependency for at least: ### Compiler / program lifecycle - parse StarLang from bounded UTF-8 bytes/string/path through the closed parser; - resolve only explicitly configured locked imports/effect ports; - expand, validate, and compile to deterministic runtime-neutral IR; - return structured diagnostics/conditions without entering the debugger; - expose compiler/build/schema identity and compiled-program digest; - support compile-only/check-only operation without starting a runtime. ### Runtime lifecycle - create an isolated runtime instance from explicit configuration; - instantiate/load a compiled StarLang program into that runtime; - start/stop/shutdown deterministically; - no implicit singleton runtime or required global registry; - multiple independent StarLang runtimes must coexist in one SBCL image; - all background threads/workers are runtime-owned and joinable on shutdown; - failures are structured conditions/results, not process exits. ### Host ports / capabilities - hosts provide explicitly typed ports/capabilities to StarLang; - no arbitrary Common Lisp function object may be embedded in `.star` source or normalized IR; - no host package names, actor refs, broker handles, database handles, credentials, or Lisp objects may leak into portable IR; - effect/adapter lookup is explicit and instance-scoped; - host callbacks, where an adapter necessarily needs one internally, remain behind a closed final-system port contract and are not StarLang language values. ### Matcher / document-transform support Add a StarLang-owned bounded matcher/transform semantic surface sufficient for `STAR-RESEARCH-051`: - typed accepted input document/schema set; - typed emitted document/schema set; - field selectors over canonical documents; - deterministic predicates/combinators; - bounded textual matching/extraction; - closed URI candidate/parse/normalize primitive suitable for HTTP(S) URL extraction; - per-input and aggregate output ceilings; - deadline/budget propagation; - typed transform failure/result; - provenance metadata describing source document, transform identity/version, selected field/location, and run/correlation identity. The matcher language must be StarLang. Do **not** expose PCRE strings plus Lisp callbacks as the canonical model. Native Common Lisp implementation details may exist inside the StarLang runtime for closed primitives, but downstream hosts invoke StarLang semantics, not those implementation functions directly. ### Downstream integration contract `starintel-server` must be able to: 1. ASDF-load only final StarLang systems; 2. compile/load approved `.star` matcher/transform definitions; 3. bind typed application ports such as canonical document ingest; 4. run matcher/transform programs in an explicit StarLang runtime instance; 5. receive typed generated-document results/provenance; 6. submit those results through the server's canonical ingest boundary; 7. replace/restart the StarLang runtime without stale Lisp/actor references surviving the generation boundary; 8. shut the server down with no leaked StarLang threads/resources. The server remains authoritative for persistence, authentication/authorization, deployment policy, runtime resource admission, and canonical ingest. StarLang remains authoritative for language semantics and matcher/transform evaluation. ## Required isolation and safety properties - bounded source bytes, parser depth/nodes/tokens, compilation work, transform input bytes, candidates, outputs, and runtime deadlines; - deterministic behavior for pure matcher/transform programs; - no Common Lisp `READ`/`EVAL` of `.star` source; - no ambient credentials or filesystem/network access; - outbound effects only through explicit capabilities/ports; - no process-global mutable matcher registry; - no raw actor refs retained by compiled programs; - runtime shutdown/restart invalidates runtime-local handles cleanly; - embedding failures cannot call `quit`, terminate SBCL, or enter an interactive debugger. ## Implementation slices ### Slice 1 — embedding API contract and RED tests - define final public packages/system dependencies; - specify runtime/compiler/program handles and structured conditions; - add fresh-process tests that load only final systems; - add two-runtime isolation tests; - add shutdown/no-leak tests; - add static guards against prototype package dependencies. ### Slice 2 — final compiler embedding path - move/finish closed parser, expansion, validation, normalized IR, loader and import/effect boundaries under final systems; - expose compile/check APIs; - satisfy the relevant #6/#47/#52 gates. ### Slice 3 — final runtime embedding path - expose create/load/start/stop/shutdown APIs around final `starlang-runtime`; - make runtime-owned resources instance-scoped; - prove multiple runtimes coexist safely. ### Slice 4 — StarLang matcher/transform semantics - implement typed matcher/transform declarations and IR; - implement bounded selectors/predicates/extraction primitives; - add HTTP(S) URI extraction fixtures; - prohibit arbitrary host callbacks/raw Lisp objects in declarations and IR. ### Slice 5 — StarIntel integration proof Using a test consumer or `starintel-server` integration branch: - load final StarLang systems only; - compile a `.star` URL matcher/transform; - feed a canonical test document; - obtain typed URL + relation output with provenance; - route output into a fake/contract canonical-ingest port; - restart runtime and prove stale references fail; - shutdown and prove no threads/resources leak. ## Acceptance gate: `STARLANG_EMBED_READY` Do not claim StarLang production-embedding readiness until all are true: - [ ] `starlang-compiler` owns the complete production compiler path. - [ ] public embedding API imports no `*.prototype` package. - [ ] normal embedding path does not load `starlang-prototype` or files under `prototype/`. - [ ] #6 conformance blockers relevant to parsing/compiler/wire contracts are closed. - [ ] #47/#52 final-system migration gates required by the embedding path are green. - [ ] two isolated runtimes can coexist in one process. - [ ] deterministic shutdown leaves no StarLang-owned thread/process/resource behind. - [ ] compiled programs contain no raw Common Lisp callbacks/objects or raw actor refs. - [ ] bounded StarLang matcher/transform semantics exist and pass URL extraction fixtures. - [ ] a downstream StarIntel integration test compiles and runs a `.star` transform through final systems only. - [ ] Nix + ASDF + CI run the same embedding test matrix from a clean environment. ## Non-goals - StarLang does not become StarIntel persistence authority. - StarLang does not own server authentication/authorization policy. - Do not add a second out-of-process language server just to embed StarLang; in-process Common Lisp embedding is the target. - Do not preserve the old server matcher callback registry as an alternate authority. - Do not declare readiness merely because `prototype/star-lang-api.lisp` can be called today. ## Definition of done This epic closes when `STARLANG_EMBED_READY` is green and a StarIntel consumer can depend only on released/final StarLang systems to compile and execute bounded StarLang programs in-process, including the matcher/transform vertical required by server #44.