StarIntel Server

• StarIntel Server
  • Position in the Quasar / StarIntel stack
  • What the server does
  • Quick start: Nix-built Compose stack
  • Submit a document
  • Build and test
  • Run from Common Lisp
  • Documentation map
  • Runtime status
  • API documentation
  • License

StarIntel Server

StarIntel Server is the Common Lisp runtime for storing, routing, querying, and recursively enriching StarIntel documents.

It combines:

• CouchDB for durable documents, views, and full-text search.
• RabbitMQ topic exchanges for document and actor traffic.
• Sento actors for local concurrency, supervision boundaries, timers, and message passing.
• A Ningle/Clack/Hunchentoot HTTP API.
• The star-cl document library and StarIntel specification adapters.
• Nix-built binaries, tests, and container images.

This repository is an experimental operator system, not a hardened public SaaS service.

Do not expose the HTTP API or RabbitMQ directly to the public Internet.

The HTTP API currently has no authentication or authorization and sends Access-Control-Allow-Origin: *. Put it behind an authenticated reverse proxy, restrict network access, and treat the Lisp init file as trusted executable code.

Position in the Quasar / StarIntel stack

StarIntel Server is a backend service layer. It is not the Quasar browser UI and it is not replaced by quasar-ui.

The intended deployment split is:

quasar-ui
  browser UI / graph renderer / standalone subset
        |
        | typed commands, projections, capability discovery
        v
quasar
  canonical Common Lisp control plane/runtime
        |
        | StarIntel service APIs and adapters
        v
starintel-server
  persistent ingest / storage / search / routing / RabbitMQ
        |
        +-----------------------------+
        |                             |
        v                             v
star-bbpd                       other actor services
  external recon actors          collectors / analyzers / tools

Responsibilities are intentionally separated:

• quasar-ui owns browser presentation, mobile/PWA behavior, Cytoscape rendering, browser-local standalone workspaces, and browser-safe bounded actions.
• quasar owns the canonical Common Lisp command/control boundary for migrated durable Quasar operations, persistent runtime supervision, privileged local integrations, reconnect/replay, and capability discovery.
• starintel-server owns the persistent StarIntel backend responsibilities implemented here: document ingest, CouchDB persistence/querying, RabbitMQ document/actor routing, HTTP service boundaries, and recursive dataflow.
• star-bbpd and similar repositories are external actor services. BBPD consumes actor-specific RabbitMQ targets, runs Subfinder, Nmap, Httpx, Katana and DNS workflows, and publishes derived StarIntel documents and relations.

A connected Quasar UI may expose controls and results for server/external services, but that does not make those capabilities browser implementations. The UI should discover which runtime/service capabilities are actually available.

Standalone quasar-ui remains valid without this server. That mode is a bounded subset and must not be documented as feature-equivalent to a connected StarIntel deployment.

What the server does

A document normally moves through this pipeline:

HTTP/client/actor
      |
      v
documents.ingest.<dtype>
      |
      v
CouchDB insert + _id/_rev enrichment
      |
      v
documents.new.<dtype>
      |
      +--> local actor via TELL
      +--> remote actor via RabbitMQ topic route
      +--> derived documents and relations
                 |
                 +--> documents.ingest.<dtype>  (durable recursion)
                 `--> documents.new.<dtype>     (event-only fan-out)

Actors can emit more StarIntel documents, relations, targets, and actor events. Those outputs can trigger more actors. This is dataflow recursion: the graph expands through messages rather than recursive function calls.

Quick start: Nix-built Compose stack

Requirements: Nix with flakes, Docker Engine, Docker Compose v2, curl, jq, and openssl.

cp .env.example .env
install -d -m 0700 secrets
openssl rand -base64 32 > secrets/couchdb_password
openssl rand -base64 48 > secrets/couchdb_secret
openssl rand -hex 24 | tr '[:lower:]' '[:upper:]' > secrets/erlang_cookie
openssl rand -base64 32 > secrets/rabbitmq_password
openssl rand -base64 32 > secrets/valkey_password
chmod 0600 secrets/*

nix run .#load-images
docker compose up --detach --wait
curl --fail http://127.0.0.1:5000/health

Default local endpoints:

Service	Address
StarIntel HTTP API	http://127.0.0.1:5000
CouchDB	http://127.0.0.1:5984
RabbitMQ AMQP	127.0.0.1:5672
RabbitMQ management	http://127.0.0.1:15672
Valkey	127.0.0.1:6379

See DOCKER.md for image builds, secrets, persistence, migration, backup, FTS, and upgrades.

Submit a document

The body must contain a dtype. The route chooses the RabbitMQ routing key but does not currently inject or validate the body dtype.

curl --fail \
  --header 'Content-Type: application/json' \
  --request POST \
  --data '{
    "_id": "example-note",
    "dataset": "demo",
    "dtype": "note",
    "sources": ["manual"],
    "version": "0.8.0",
    "dateAdded": 0,
    "dateUpdated": 0,
    "content": "first document"
  }' \
  http://127.0.0.1:5000/new/document/note

The API acknowledges queue publication, not CouchDB persistence. Read the document after the ingest consumer has processed it:

curl --fail http://127.0.0.1:5000/document/example-note | jq

Build and test

nix build
nix run .#star-unit-tests

With CouchDB and RabbitMQ available:

nix run .#star-integration-tests

Full Nix image, health, FTS, restart, and persistence test:

./scripts/stack-test.sh

See doc/testing.md.

Run from Common Lisp

The pinned Nix build is the supported reproducible path. For interactive development:

nix develop
sbcl --load run.lisp

Build the executable:

nix build
./result/bin/star-server start --init ./example_configs/init.lisp

The executable accepts:

star-server start -i PATH
star-server start --init PATH

The same path can be supplied through STAR_SERVER_INIT_FILE.

Documentation map

Document	Contents
doc/index.org	Documentation index and implementation status
doc/architecture.org	Runtime structure, startup order, concurrency, and repository layout
doc/actors.org	Creating, registering, targeting, scheduling, and operating actors
doc/messaging.org	RabbitMQ exchanges, queues, routing keys, recursion, delivery, and loop control
doc/document-spec.org	StarIntel 0.9 and legacy 0.8 documents, types, relations, IDs, metadata, and provenance
doc/configuration.org	Environment, init files, secrets, advanced examples, and tuning
doc/http-api-docs.org	HTTP endpoint reference and examples
doc/lease-store-usage.org	Backend-neutral target lease protocol examples and lifecycle
DOCKER.md	Nix-built container stack and operations
doc/testing.md	Unit, integration, and stack tests

Runtime status

The documentation distinguishes three states:

• Active: loaded by source/starintel-gserver.asd and started by star::main.
• Present but not active: code exists in the repository but is not loaded by the ASDF system or is not started by the current startup path.
• Stub/experimental: API or actor code exists but is incomplete.

Important current limits:

• HTTP ingestion does not enforce the strict StarIntel 0.9 schema.
• The server still uses legacy flat 0.8 constructors in parts of the actor code.
• source/actor-systems/user-finder.lisp and user-hunt.lisp are not loaded by the ASDF system.
• The matcher actor framework is experimental; the URL extractor is loaded, but a complete global pattern-dispatch loop is not wired.
• /new/event/:id is a stub.
• *http-api-base-path* and the HTTP certificate/key variables are not applied by start-http-api.
• The second /dataset-size route definition replaces or shadows the first, depending on Ningle route behavior.

These are documented facts, not supported guarantees.

API documentation

The exported API of every package is documented where it is defined: as org-mode markup inside Common Lisp docstrings. Render it into Org files under doc/api/ with:

make docs-api        # = nix run .#gen-api-docs

This runs the org-doc extractor against the loaded starintel-gserver image and regenerates doc/api/index.org plus one file per package. Docstring coverage is printed at the end of the run so undocumented symbols are visible.

In Emacs, install the sly-org-doc package (from the org-doc repo) to see docstring summaries in the echo area while typing (SLY and SLIME), and C-c C-d C-o to open the full Org docstring of the symbol at point, fontified and live.

License

See LICENSE.