observability: StarIntel telemetry abstraction + OTLP instrumentation (addon, opt-in) #9

Merged

nsaspy merged 2 commits from feat/observability-slice into master 2026-09-11 02:52:00 +00:00

Conversation 0 Commits 2 Files changed 32 +1633 -18

nsaspy commented 2026-09-11 02:46:18 +00:00

Owner

Copy link

Closes #8 (server side of the observability slice; infra slice merged in starintel-labs/starintel-infra#163).

What

New ASDF system =starintel-observability= (package =star.observability=) — the StarIntel observability abstraction. Applications depend on this, never on an OpenObserve client.

h2. Lifecycle: opt-in via init.lisp (per review decision)

• The addon is registered at gserver load (metadata only, the addon-design contract).
• Nothing runs until the operator adds =(load-addon :starintel-observability)= to the trusted init file (documented in =example_configs/init.lisp= and doc/configuration.org).
• Without that line: no exporter thread, no queues, no ids, no counters (proven by the =disabled-addon-costs-nothing= FiveAM test). =start-runtime= contains no hard-coded telemetry wiring.
• =STAR_OBSERVABILITY_ENABLED=false= kill switch overrides everything, and =STAR_OBSERVABILITY_SIGNALS= selects logs/metrics/traces.

h2. Instrumentation

• HTTP: outermost lack middleware — extracts/injects W3C =traceparent=, one =http.server= span per request, =starintel_http_requests_total= + duration with route templates matched from the HTTP contract table (raw URLs never become labels).
• RabbitMQ: publish choke-point injects traceparent into AMQP headers (never bodies) and counts =starintel_rabbit_publish_total= / =_failures_total=; consumer around-method joins the producer's trace from captured AMQP properties and records consume metrics.
• CouchDB: instrumented view transport (installed at addon start; the test swap seam =couchdb-view-transport= stays intact) with client spans and =starintel_couchdb_requests_total=; document bodies never recorded.
• Leases: =observability-lease-metrics-hook= for the existing lease =metrics-hook= seams, including =starintel_lease_stale_writer_rejections_total= (store wiring itself is a documented gap — no runtime lease store is adopted yet).
• Security events: authn ok/failed, authz denied, tenant-boundary rejection as structured =security.event= logs. Never credentials.
• Structured logging: =emit-log-event= JSON events with =event.name= + trace correlation; log4cl human logs unchanged.

h2. Redaction

Layer 1 at emission: structural =redact-attributes= deletes the forbidden-key namespace (authorization/cookie/password/secret/token/api_key/access_token/refresh_token/client_secret/bodies), case-insensitive. Collector (transform/redact) and OpenObserve ingestion are layers 2/3. =STAR_INCLUDE_PAYLOADS= is always off unless an explicit debug policy sets it.

h2. Compose

=otelcol= sidecar (otelcol-contrib): receives app OTLP, scrapes rabbitmq 15692 + couchdb 17986 where enabled, forwards to the observe gateway with no OpenObserve credentials (the gateway owns the ingest token).

h2. Tests (hermetic FiveAM, =nix run .#star-unit-tests=)

Traceparent round-trip + malformed rejection; disabled-mode no-op; structural redaction (case-insensitive); backend-failure → bounded drop counters without signaling; HTTP→nested-span correlation through a captured export transport (one trace id end-to-end, =TEST_API_SECRET_DO_NOT_LEAK_123= never present, span count per request bounded).

Validation: full unit suite green (23 suites), =nix flake check= green, doc-coverage 844/844, prolog-verify check pass at the exact head.

Closes #8 (server side of the observability slice; infra slice merged in starintel-labs/starintel-infra#163). ## What New ASDF system =starintel-observability= (package =star.observability=) — the StarIntel observability abstraction. Applications depend on this, never on an OpenObserve client. h2. Lifecycle: opt-in via init.lisp (per review decision) - The addon is *registered* at gserver load (metadata only, the addon-design contract). - Nothing runs until the operator adds =(load-addon :starintel-observability)= to the trusted init file (documented in =example_configs/init.lisp= and doc/configuration.org). - Without that line: no exporter thread, no queues, no ids, no counters (proven by the =disabled-addon-costs-nothing= FiveAM test). =start-runtime= contains no hard-coded telemetry wiring. - =STAR_OBSERVABILITY_ENABLED=false= kill switch overrides everything, and =STAR_OBSERVABILITY_SIGNALS= selects logs/metrics/traces. h2. Instrumentation - *HTTP*: outermost lack middleware — extracts/injects W3C =traceparent=, one =http.server= span per request, =starintel_http_requests_total= + duration with route templates matched from the HTTP contract table (raw URLs never become labels). - *RabbitMQ*: publish choke-point injects traceparent into AMQP headers (never bodies) and counts =starintel_rabbit_publish_total= / =_failures_total=; consumer around-method joins the producer's trace from captured AMQP properties and records consume metrics. - *CouchDB*: instrumented view transport (installed at addon start; the test swap seam =*couchdb-view-transport*= stays intact) with client spans and =starintel_couchdb_requests_total=; document bodies never recorded. - *Leases*: =observability-lease-metrics-hook= for the existing lease =metrics-hook= seams, including =starintel_lease_stale_writer_rejections_total= (store wiring itself is a documented gap — no runtime lease store is adopted yet). - *Security events*: authn ok/failed, authz denied, tenant-boundary rejection as structured =security.event= logs. Never credentials. - *Structured logging*: =emit-log-event= JSON events with =event.name= + trace correlation; log4cl human logs unchanged. h2. Redaction Layer 1 at emission: structural =redact-attributes= deletes the forbidden-key namespace (authorization/cookie/password/secret/token/api_key/access_token/refresh_token/client_secret/bodies), case-insensitive. Collector (transform/redact) and OpenObserve ingestion are layers 2/3. =STAR_INCLUDE_PAYLOADS= is always off unless an explicit debug policy sets it. h2. Compose =otelcol= sidecar (otelcol-contrib): receives app OTLP, scrapes rabbitmq 15692 + couchdb 17986 where enabled, forwards to the observe gateway with **no OpenObserve credentials** (the gateway owns the ingest token). h2. Tests (hermetic FiveAM, =nix run .#star-unit-tests=) Traceparent round-trip + malformed rejection; disabled-mode no-op; structural redaction (case-insensitive); backend-failure → bounded drop counters without signaling; HTTP→nested-span correlation through a captured export transport (one trace id end-to-end, =TEST_API_SECRET_DO_NOT_LEAK_123= never present, span count per request bounded). Validation: full unit suite green (23 suites), =nix flake check= green, doc-coverage 844/844, prolog-verify check pass at the exact head.

nsaspy added 1 commit 2026-09-11 02:46:18 +00:00

observability: StarIntel telemetry abstraction + OTLP instrumentation ... 87622e0c9d

New ASDF system starintel-observability (package star.observability):
- OTLP/HTTP-JSON exporter (dexador) with bounded per-signal queues,
  background flush, wall-clock timeout wrapper, drop-with-counter on
  failure; injectable *export-batch-fn* for hermetic tests
- W3C traceparent/tracestate parse+encode; HTTP middleware (outermost
  lack boundary) with trace extraction, server span, request metrics and
  traceparent response injection; route templates from the HTTP contract
  table keep metric labels bounded
- structural redaction at emission (forbidden-key namespace) plus metric
  label cardinality guard (starintel IDs never become dimensions)
- RabbitMQ trace propagation via AMQP headers at the publish choke-point
  and consumer around-method; CouchDB view transport wrapper; lease
  metrics hook (starintel_lease_stale_writer_rejections_total)
- security events (authn ok/failed, authz denied, tenant boundary) and
  structured event.name log records with trace correlation

Lifecycle: addon-registered, metadata-only at load; operators opt in
with (load-addon :starintel-observability) in init.lisp. Without it no
exporter thread, queues, ids, or counters exist (test-verified); the
STAR_OBSERVABILITY_ENABLED=false kill switch overrides everything.

Compose: otelcol-contrib sidecar with no OpenObserve credentials,
forwarding to the observe gateway and scraping rabbitmq/couchdb.

Tests: hermetic FiveAM suite (traceparent round-trip, redaction,
backend-failure drops, HTTP correlation with captured transport, volume
bound). Full unit suite green; doc-coverage 844/844.

nsaspy added 1 commit 2026-09-11 02:49:09 +00:00

docs: regenerate API reference for the observability systems bc64bd02fd

nsaspy merged commit 15a4171028 into master 2026-09-11 02:52:00 +00:00

nsaspy referenced this pull request from a commit 2026-09-11 02:52:02 +00:00

Merge pull request 'observability: StarIntel telemetry abstraction + OTLP instrumentation (addon, opt-in)' (#9) from feat/observability-slice into master

nsaspy referenced this pull request 2026-09-11 02:52:09 +00:00

observability: StarIntel telemetry abstraction + OTLP instrumentation (PR2 server) #8

nsaspy referenced this pull request 2026-09-11 08:38:24 +00:00

leases: wire the Valkey lease store into the runtime with the observability metrics hook #11

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starintel-labs/starintel-server!9

No description provided.