observability: StarIntel telemetry abstraction + OTLP instrumentation (PR2 server) #8

New issue

Closed

opened 2026-09-10 20:57:03 +00:00 by nsaspy · 2 comments

nsaspy commented

2026-09-10 20:57:03 +00:00

Owner

Implementation issue for the starintel-server side of the OpenObserve observability slice. Umbrella + infra slice: starintel-labs/starintel-infra#162.

h2. Scope (PR 2 — starintel-server feat/observability-slice)

New ASDF system =starintel-observability= (package =star.observability=, addon-registered via =register-addon=) implementing the StarIntel observability abstraction. Applications depend on this, never on an OpenObserve client. Telemetry flows:

star-server (OTLP/HTTP-JSON, no credentials)
  -> otelcol sidecar (docker compose, no credentials)
  -> observe.star.intel gateway collector (OTLP 4317/4318)
  -> OpenObserve

h2. Contents

Config (env knobs, gserver-settings.lisp pattern + doc/configuration.org table): =STAR_OBSERVABILITY_ENABLED= (default t per locked decision; prod compose sets explicitly), =STAR_OBSERVABILITY_ENDPOINT= (default =http://observe.star.intel:4318=; compose points at sidecar), =STAR_OBSERVABILITY_SIGNALS=logs,metrics,traces=, =STAR_TRACE_SAMPLE_RATE=0.05=, =STAR_ACTOR_DETAIL=normal=, =STAR_PROLOG_DETAIL=goal=, =STAR_INCLUDE_PAYLOADS= (always false unless an explicit debug policy is set). Disabled => no exporter thread, no network calls, fully functional app.
OTLP exporter: OTLP/HTTP-JSON (jsown encode, dexador POST) to =/v1/logs|traces|metrics=; batched background flush thread; bounded queue; drop-with-counter on overflow/failure; short timeouts; export failures rate-limited WARN logs. Delta temporality for counters.
Trace context: W3C =traceparent=/=tracestate= parse+generate. HTTP middleware extracts/injects. RabbitMQ publish injects =traceparent= into AMQP headers (never bodies); consumers extract + child spans. Target-dispatch envelope =trace-id= unified to W3C. CouchDB spans at =couchdb-view-transport= + call-site wrappers (operation, database, status, duration, retry, conflict — never document bodies). Prolog: =prolog.goal=/=prolog.phase= span helpers + designed-but-unimplemented =rlm.run/=spec/=plan/=execute/=verify hierarchy (no Prolog boundary exists in the server yet — honest gap). Predicate-level tracing only behind explicit debug mode.
Instrumentation: HTTP (route templates from cli/http-contract.lisp operation table — no raw-URL labels; method, status class, latency, active, 4xx/5xx, authn-failure, authz-deny), actors (actor.spawn/send/receive/handle/reply/restart/stop spans at Sento seams; metrics: active, sent, consumed, handler duration, failures, restarts; no payload attributes), RabbitMQ app-level (publish/consume counts + failures), leases (wire existing metrics-hook: =starintel_lease_stale_writer_rejections_total=, conflicts, outbox age), security events (authn ok/fail, invalid api key, authz allow/deny, tenant-boundary rejection — never keys/tokens/headers/cookies), structured =event.name= JSON log events on OTLP logs with trace correlation.
Redaction at emission: structural field policy + denylist scrub (authorization, cookie, set-cookie, password, secret, token, api_key, access_token, refresh_token, client_secret, request/response/document/message bodies). Collector + OpenObserve ingestion are layers 2/3.
LLM telemetry contract: model, provider, latency, tokens in/out/cached, cost estimate, retries, run/operation id; content capture explicitly disabled by default.
Compose: =otelcol= sidecar service (otelcol-contrib, no OpenObserve credentials): receives app OTLP, scrapes rabbitmq prometheus (15692) + couchdb prometheus (17986) where reachable, ships to observe gateway. =STAR_OBSERVABILITY_ENDPOINT=otelcol:4318=.

h2. Acceptance criteria (PR 2)

Disabled mode: no OTLP endpoint configured => app fully functional, zero network telemetry attempts (test proves).
One distributed trace visible across HTTP -> RabbitMQ -> actor -> CouchDB via fake-collector test harness.
=TEST_API_SECRET_DO_NOT_LEAK_123= / =TEST_OAUTH_TOKEN_DO_NOT_LEAK_456= injected through representative paths are absent from logs, traces, metrics, and exporter diagnostics.
Cardinality guard: tenant/actor/target/document IDs, raw URLs, usernames never become unbounded metric labels.
Volume regression: single request produces a bounded span count.
Backend unavailable: exporter drops with counters, app stays healthy.
All new exports carry docstrings (doc-coverage CI gate). FiveAM suite green via =nix run .#star-unit-tests=.
Credential model preserved: star-server and all StarIntel apps hold no OpenObserve credentials.

h2. Out of scope

Predicate-level Prolog tracing by default; LLM content capture; alert routing beyond the webhook boundary contract; OpenObserve HA.

*Implementation issue for the starintel-server side of the OpenObserve observability slice. Umbrella + infra slice: starintel-labs/starintel-infra#162.* h2. Scope (PR 2 — starintel-server `feat/observability-slice`) New ASDF system =starintel-observability= (package =star.observability=, addon-registered via =register-addon=) implementing the StarIntel observability abstraction. Applications depend on this, never on an OpenObserve client. Telemetry flows: <pre> star-server (OTLP/HTTP-JSON, no credentials) -> otelcol sidecar (docker compose, no credentials) -> observe.star.intel gateway collector (OTLP 4317/4318) -> OpenObserve </pre> h2. Contents - *Config* (env knobs, gserver-settings.lisp pattern + doc/configuration.org table): =STAR_OBSERVABILITY_ENABLED= (default **t** per locked decision; prod compose sets explicitly), =STAR_OBSERVABILITY_ENDPOINT= (default =http://observe.star.intel:4318=; compose points at sidecar), =STAR_OBSERVABILITY_SIGNALS=logs,metrics,traces=, =STAR_TRACE_SAMPLE_RATE=0.05=, =STAR_ACTOR_DETAIL=normal=, =STAR_PROLOG_DETAIL=goal=, =STAR_INCLUDE_PAYLOADS= (**always** false unless an explicit debug policy is set). Disabled => no exporter thread, no network calls, fully functional app. - *OTLP exporter*: OTLP/HTTP-JSON (jsown encode, dexador POST) to =/v1/logs|traces|metrics=; batched background flush thread; bounded queue; drop-with-counter on overflow/failure; short timeouts; export failures rate-limited WARN logs. Delta temporality for counters. - *Trace context*: W3C =traceparent=/=tracestate= parse+generate. HTTP middleware extracts/injects. RabbitMQ publish injects =traceparent= into AMQP **headers** (never bodies); consumers extract + child spans. Target-dispatch envelope =trace-id= unified to W3C. CouchDB spans at =*couchdb-view-transport*= + call-site wrappers (operation, database, status, duration, retry, conflict — never document bodies). Prolog: =prolog.goal=/=prolog.phase= span helpers + designed-but-unimplemented =rlm.run/=spec/=plan/=execute/=verify hierarchy (no Prolog boundary exists in the server yet — honest gap). Predicate-level tracing only behind explicit debug mode. - *Instrumentation*: HTTP (route templates from cli/http-contract.lisp operation table — no raw-URL labels; method, status class, latency, active, 4xx/5xx, authn-failure, authz-deny), actors (actor.spawn/send/receive/handle/reply/restart/stop spans at Sento seams; metrics: active, sent, consumed, handler duration, failures, restarts; no payload attributes), RabbitMQ app-level (publish/consume counts + failures), leases (wire existing metrics-hook: =starintel_lease_stale_writer_rejections_total=, conflicts, outbox age), security events (authn ok/fail, invalid api key, authz allow/deny, tenant-boundary rejection — never keys/tokens/headers/cookies), structured =event.name= JSON log events on OTLP logs with trace correlation. - *Redaction at emission*: structural field policy + denylist scrub (authorization, cookie, set-cookie, password, secret, token, api_key, access_token, refresh_token, client_secret, request/response/document/message bodies). Collector + OpenObserve ingestion are layers 2/3. - *LLM telemetry contract*: model, provider, latency, tokens in/out/cached, cost estimate, retries, run/operation id; content capture explicitly disabled by default. - *Compose*: =otelcol= sidecar service (otelcol-contrib, no OpenObserve credentials): receives app OTLP, scrapes rabbitmq prometheus (15692) + couchdb prometheus (17986) where reachable, ships to observe gateway. =STAR_OBSERVABILITY_ENDPOINT=otelcol:4318=. h2. Acceptance criteria (PR 2) 1. Disabled mode: no OTLP endpoint configured => app fully functional, zero network telemetry attempts (test proves). 2. One distributed trace visible across HTTP -> RabbitMQ -> actor -> CouchDB via fake-collector test harness. 3. =TEST_API_SECRET_DO_NOT_LEAK_123= / =TEST_OAUTH_TOKEN_DO_NOT_LEAK_456= injected through representative paths are absent from logs, traces, metrics, and exporter diagnostics. 4. Cardinality guard: tenant/actor/target/document IDs, raw URLs, usernames never become unbounded metric labels. 5. Volume regression: single request produces a bounded span count. 6. Backend unavailable: exporter drops with counters, app stays healthy. 7. All new exports carry docstrings (doc-coverage CI gate). FiveAM suite green via =nix run .#star-unit-tests=. 8. Credential model preserved: star-server and all StarIntel apps hold no OpenObserve credentials. h2. Out of scope Predicate-level Prolog tracing by default; LLM content capture; alert routing beyond the webhook boundary contract; OpenObserve HA.

nsaspy commented

2026-09-10 20:57:30 +00:00

Author

Owner

Umbrella + infra slice: starintel-labs/starintel-infra#162. Cross-linked; see infra issue for design decisions and acceptance criteria.

nsaspy referenced this issue from a pull request that will close it,

2026-09-11 02:46:18 +00:00

observability: StarIntel telemetry abstraction + OTLP instrumentation (addon, opt-in) #9

nsaspy closed this issue

2026-09-11 02:52:00 +00:00

nsaspy commented

2026-09-11 02:52:09 +00:00

Author

Owner

PR 2 merged: #9 (feat/observability-slice, 87622e0 + bc64bd0). Local gates at exact head: star-unit-tests 23/23 suites green (incl. 8 observability tests), nix flake check green, doc-coverage 844/844, gen-api-docs clean, prolog-verify check pass. NOTE: the doc-coverage CI job fails identically on master (pre-existing since 2065321; runs 3037-3401 all failure, job logs 404) - not a regression from this PR; filed as follow-up infra investigation.

PR 2 merged: #9 (feat/observability-slice, 87622e0 + bc64bd0). Local gates at exact head: star-unit-tests 23/23 suites green (incl. 8 observability tests), nix flake check green, doc-coverage 844/844, gen-api-docs clean, prolog-verify check pass. NOTE: the doc-coverage CI job fails identically on master (pre-existing since 2065321; runs 3037-3401 all failure, job logs 404) - not a regression from this PR; filed as follow-up infra investigation.

nsaspy referenced this issue from a commit

2026-09-11 08:37:49 +00:00

leases: wire the Valkey lease store into the runtime with the observability metrics hook

nsaspy referenced this issue

2026-09-11 08:38:24 +00:00

leases: wire the Valkey lease store into the runtime with the observability metrics hook #11

nsaspy referenced this issue from a commit

2026-09-11 08:40:36 +00:00

Merge pull request 'leases: wire the Valkey lease store into the runtime with the observability metrics hook' (#11) from feat/valkey-lease-runtime into master