Add working .star loader, Lisp API, and Star-CL constructors #37

Merged

lost-rob0t merged 40 commits from agent/star-loader-url-import-star-cl-port into main

2026-07-24 02:08:57 +00:00

lost-rob0t commented

2026-07-23 23:47:48 +00:00

(Migrated from github.com)

What changed

Lisp-first `.star` loader

The loader is a Common Lisp API. The shell command is now only a thin adapter over star-lang.api.

Public Lisp entry points include:

star-lang.api:load-star
star-lang.api:load-star-file
star-lang.api:load-star-url
star-lang.api:load-star-runtime

(load "prototypes/star-lang/common-lisp/star-lang-api.lisp")

(defparameter *graph*
  (star-lang.api:load-star-file "root.star"))

The loader safely reads one bounded top-level Star-Lang form, disables reader evaluation and dispatch syntax, recursively resolves local and HTTP(S) imports, verifies exact library identity and SHA-256 locks, detects cycles and conflicting identities, caches verified remote specifications, and supports offline cache reuse.

The CLI delegates to the same API:

prototypes/star-lang/bin/star load root.star
prototypes/star-lang/bin/star load root.star --allow-network
prototypes/star-lang/bin/star load root.star --cache ~/.cache/star-lang/specs --manifest resolved.sexp

Standalone Star-CL schema

add org.starintel/star-cl@1 as the canonical Star-Lang representation of Star-CL
port every existing Star-CL document kind
add the intended port and asn document types referenced by legacy constructors but absent from the original class definitions
expand documents with applicable provenance, collection, temporal, confidence, verification, custody, ownership, access, retention, content, lifecycle, normalization, jurisdiction, and extension metadata
add artifact, finding, scope, and runtime-event contracts

ID policies and API

Documents may declare:

ULID
UUIDv4
deterministic MD5
deterministic SHA-256
supplied IDs

The Lisp API exports make-ulid, make-uuidv4, make-digest-id, and generate-id. Legacy Star-CL types retain deterministic MD5 identity where applicable, general entities and relations use ULIDs, content-addressed records use SHA-256, and runtime events demonstrate UUIDv4.

Generated constructor functions

Add a constructor generator that reads declarative constructor metadata from loaded .star libraries and installs real Common Lisp defuns into a selected package.

(defparameter *star-cl*
  (star-lang.api:load-star-runtime
   "prototypes/star-lang/fixtures/star-cl-constructors.star"
   :constructor-package "STARINTEL"))

(starintel:new-person
 "people" "Ada" "Lovelace" "person"
 :bio "Mathematician")

(starintel:new-relation
 "relations" source-id target-id
 :predicate "member-of")

The generator preserves old Star-CL constructor semantics and signatures, including:

positional arguments for new-person and new-org
sparse (dataset &rest args) constructors
new-email* local-part/domain splitting
new-relation keyword defaults and the legacy predicate allowlist
new-target keyword defaults
new-target-without-options
the intended new-username function that was exported by old Star-CL but had no implementation

Explicit compatibility constructors disable strict required-field validation to retain old sparse-construction behavior. Automatically generated constructors for other document types remain strict and use (dataset &rest args).

Constructor source can be emitted without installing it:

(with-open-file (stream "star-cl-constructors.lisp"
                        :direction :output
                        :if-exists :supersede
                        :if-does-not-exist :create)
  (star-lang.api:generate-constructor-source
   *star-cl* stream :package "STARINTEL"))

Document runtime

The runtime resolves inherited contracts, applies defaults and shared metadata, generates IDs from document policies, validates typed fields, supports CouchDB _id and _rev, encodes camelCase/snake_case/native field names, decodes external keys case-insensitively, and creates typed relation documents.

Primary functions:

compile-document-contract
create-document
set-document-meta
validate-document
encode-document
decode-document
relate-documents
document-value

Validation

Four workflow tracks pass:

Star-Lang Constructors — form balance, exact legacy signatures, sparse construction, email parsing, target defaults, relation validation, automatic constructors, source generation, and verification that the CLI delegates to the Lisp API
Star-Lang Document Runtime — ULID, UUIDv4, MD5, SHA-256, policy inheritance, typed construction, metadata, validation, CouchDB/camelCase round trips, and relation creation
Star-Lang Loader — loader conformance, CLI operation, real HTTP import, exact digest verification, cache population, server shutdown, and offline cache reuse
Star-Lang Common Lisp Research — every existing compiler, specification, semantic, binding, lifecycle, dispatcher, execution, transport, and cl-gserver adapter suite

Deliberate boundaries

star-cl.star remains the canonical document schema. Constructor compatibility metadata currently lives in a digest-locked overlay, star-cl-constructors.star, which imports it. Consolidating that metadata into the canonical file is a later cleanup.
The generator creates constructor functions over document-instance values; it does not yet generate CLOS classes, slot accessors, or make-instance compatibility.
Encode/decode operates on Lisp map/alist values and CouchDB naming conventions; JSON text parsing/emission is not included.
CouchDB transport and persistence remain adapter work.
MD5 and SHA-256 currently invoke system md5sum and sha256sum; URL retrieval currently invokes system curl.

## What changed ### Lisp-first `.star` loader The loader is a Common Lisp API. The shell command is now only a thin adapter over `star-lang.api`. Public Lisp entry points include: - `star-lang.api:load-star` - `star-lang.api:load-star-file` - `star-lang.api:load-star-url` - `star-lang.api:load-star-runtime` ```lisp (load "prototypes/star-lang/common-lisp/star-lang-api.lisp") (defparameter *graph* (star-lang.api:load-star-file "root.star")) ``` The loader safely reads one bounded top-level Star-Lang form, disables reader evaluation and dispatch syntax, recursively resolves local and HTTP(S) imports, verifies exact library identity and SHA-256 locks, detects cycles and conflicting identities, caches verified remote specifications, and supports offline cache reuse. The CLI delegates to the same API: ```sh prototypes/star-lang/bin/star load root.star prototypes/star-lang/bin/star load root.star --allow-network prototypes/star-lang/bin/star load root.star --cache ~/.cache/star-lang/specs --manifest resolved.sexp ``` ### Standalone Star-CL schema - add `org.starintel/star-cl@1` as the canonical Star-Lang representation of Star-CL - port every existing Star-CL document kind - add the intended `port` and `asn` document types referenced by legacy constructors but absent from the original class definitions - expand documents with applicable provenance, collection, temporal, confidence, verification, custody, ownership, access, retention, content, lifecycle, normalization, jurisdiction, and extension metadata - add artifact, finding, scope, and runtime-event contracts ### ID policies and API Documents may declare: - ULID - UUIDv4 - deterministic MD5 - deterministic SHA-256 - supplied IDs The Lisp API exports `make-ulid`, `make-uuidv4`, `make-digest-id`, and `generate-id`. Legacy Star-CL types retain deterministic MD5 identity where applicable, general entities and relations use ULIDs, content-addressed records use SHA-256, and runtime events demonstrate UUIDv4. ### Generated constructor functions Add a constructor generator that reads declarative constructor metadata from loaded `.star` libraries and installs real Common Lisp `defun`s into a selected package. ```lisp (defparameter *star-cl* (star-lang.api:load-star-runtime "prototypes/star-lang/fixtures/star-cl-constructors.star" :constructor-package "STARINTEL")) (starintel:new-person "people" "Ada" "Lovelace" "person" :bio "Mathematician") (starintel:new-relation "relations" source-id target-id :predicate "member-of") ``` The generator preserves old Star-CL constructor semantics and signatures, including: - positional arguments for `new-person` and `new-org` - sparse `(dataset &rest args)` constructors - `new-email*` local-part/domain splitting - `new-relation` keyword defaults and the legacy predicate allowlist - `new-target` keyword defaults - `new-target-without-options` - the intended `new-username` function that was exported by old Star-CL but had no implementation Explicit compatibility constructors disable strict required-field validation to retain old sparse-construction behavior. Automatically generated constructors for other document types remain strict and use `(dataset &rest args)`. Constructor source can be emitted without installing it: ```lisp (with-open-file (stream "star-cl-constructors.lisp" :direction :output :if-exists :supersede :if-does-not-exist :create) (star-lang.api:generate-constructor-source *star-cl* stream :package "STARINTEL")) ``` ### Document runtime The runtime resolves inherited contracts, applies defaults and shared metadata, generates IDs from document policies, validates typed fields, supports CouchDB `_id` and `_rev`, encodes camelCase/snake_case/native field names, decodes external keys case-insensitively, and creates typed relation documents. Primary functions: - `compile-document-contract` - `create-document` - `set-document-meta` - `validate-document` - `encode-document` - `decode-document` - `relate-documents` - `document-value` ### Validation Four workflow tracks pass: - **Star-Lang Constructors** — form balance, exact legacy signatures, sparse construction, email parsing, target defaults, relation validation, automatic constructors, source generation, and verification that the CLI delegates to the Lisp API - **Star-Lang Document Runtime** — ULID, UUIDv4, MD5, SHA-256, policy inheritance, typed construction, metadata, validation, CouchDB/camelCase round trips, and relation creation - **Star-Lang Loader** — loader conformance, CLI operation, real HTTP import, exact digest verification, cache population, server shutdown, and offline cache reuse - **Star-Lang Common Lisp Research** — every existing compiler, specification, semantic, binding, lifecycle, dispatcher, execution, transport, and cl-gserver adapter suite ## Deliberate boundaries - `star-cl.star` remains the canonical document schema. Constructor compatibility metadata currently lives in a digest-locked overlay, `star-cl-constructors.star`, which imports it. Consolidating that metadata into the canonical file is a later cleanup. - The generator creates constructor functions over `document-instance` values; it does not yet generate CLOS classes, slot accessors, or `make-instance` compatibility. - Encode/decode operates on Lisp map/alist values and CouchDB naming conventions; JSON text parsing/emission is not included. - CouchDB transport and persistence remain adapter work. - MD5 and SHA-256 currently invoke system `md5sum` and `sha256sum`; URL retrieval currently invokes system `curl`.