Observability: optional OpenObserve/OTLP support via init configuration

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

nsaspy/starintel-server#128

lost-rob0t commented

2026-08-26 07:45:37 +00:00

(Migrated from github.com)

Goal

Add first-class operator-controlled observability that can export StarIntel Server logs, metrics, and traces to OpenObserve through OpenTelemetry/OTLP.

This is observability for the operator's own deployment, not StarIntel product telemetry.

OpenObserve supports OTLP logs, metrics, and traces. Prefer the OTLP contract so the server is not tightly coupled to OpenObserve-specific APIs.

Docs: https://openobserve.ai/docs/ingestion/
OTLP logs: https://openobserve.ai/docs/ingestion/logs/otlp/

Configuration boundary

Use the existing StarIntel init-file boundary (--init, STAR_SERVER_INIT_FILE, or ./init.lisp). The default configuration must preserve today's behavior.

Example shape (exact API to be designed in-repo):

(observability:configure
 :enabled nil
 :protocol :otlp-http
 :endpoint "https://observe.example/api/default"
 :stream "starintel-server"
 :signals '(:logs :metrics :traces)
 :authorization-env "STARINTEL_OTLP_AUTH")

Secrets/credentials MUST NOT be committed to or required inside the init file. The init file should reference environment/secret sources.

Required behavior

No observability config means disabled.
Disabled means no exporter initialization, no outbound network traffic, and no new required service/dependency at startup.
OpenObserve outage/export failure is non-fatal to StarIntel Server.
Export is bounded/non-blocking; define queue, batching, timeout, retry, and drop behavior.
Support individually enabling logs, metrics, and traces.
Emit standard resource identity (service.name, version, environment, instance/actor identity where appropriate).
Correlate logs with trace/span IDs where available.
Never export credentials, authorization headers, raw secrets, or sensitive request material by default.
Add redaction/filter hooks before export.
Unit tests prove zero outbound traffic when disabled.
Integration test proves configured OTLP endpoint/headers/stream are used.
Failure test proves an unreachable exporter cannot crash or stall the service.
Document an OpenObserve example and a generic OTLP example.

Architecture constraint

Keep the instrumentation/provider boundary generic. OpenObserve should be a supported OTLP destination, not a hard-coded runtime dependency.

Explicit exclusion

Do not add anonymous usage analytics, crash reporting to a StarIntel-owned endpoint, installation IDs, or any other product telemetry in this issue. Product telemetry requires a separate opt-in/privacy design decision.

## Goal Add first-class operator-controlled observability that can export StarIntel Server logs, metrics, and traces to OpenObserve through OpenTelemetry/OTLP. This is **observability for the operator's own deployment**, not StarIntel product telemetry. OpenObserve supports OTLP logs, metrics, and traces. Prefer the OTLP contract so the server is not tightly coupled to OpenObserve-specific APIs. Docs: https://openobserve.ai/docs/ingestion/ OTLP logs: https://openobserve.ai/docs/ingestion/logs/otlp/ ## Configuration boundary Use the existing StarIntel init-file boundary (`--init`, `STAR_SERVER_INIT_FILE`, or `./init.lisp`). The default configuration must preserve today's behavior. Example shape (exact API to be designed in-repo): ```lisp (observability:configure :enabled nil :protocol :otlp-http :endpoint "https://observe.example/api/default" :stream "starintel-server" :signals '(:logs :metrics :traces) :authorization-env "STARINTEL_OTLP_AUTH") ``` Secrets/credentials MUST NOT be committed to or required inside the init file. The init file should reference environment/secret sources. ## Required behavior - [ ] No observability config means disabled. - [ ] Disabled means no exporter initialization, no outbound network traffic, and no new required service/dependency at startup. - [ ] OpenObserve outage/export failure is non-fatal to StarIntel Server. - [ ] Export is bounded/non-blocking; define queue, batching, timeout, retry, and drop behavior. - [ ] Support individually enabling logs, metrics, and traces. - [ ] Emit standard resource identity (`service.name`, version, environment, instance/actor identity where appropriate). - [ ] Correlate logs with trace/span IDs where available. - [ ] Never export credentials, authorization headers, raw secrets, or sensitive request material by default. - [ ] Add redaction/filter hooks before export. - [ ] Unit tests prove zero outbound traffic when disabled. - [ ] Integration test proves configured OTLP endpoint/headers/stream are used. - [ ] Failure test proves an unreachable exporter cannot crash or stall the service. - [ ] Document an OpenObserve example and a generic OTLP example. ## Architecture constraint Keep the instrumentation/provider boundary generic. OpenObserve should be a supported OTLP destination, not a hard-coded runtime dependency. ## Explicit exclusion Do **not** add anonymous usage analytics, crash reporting to a StarIntel-owned endpoint, installation IDs, or any other product telemetry in this issue. Product telemetry requires a separate opt-in/privacy design decision.

nsaspy referenced this issue

2026-09-04 14:37:38 +00:00

[P1-08] Add structured metrics, tracing, audit events, and operational diagnostics #46

Rows
Columns

Observability: optional OpenObserve/OTLP support via init configuration #128

Goal

Configuration boundary

Required behavior

Architecture constraint

Explicit exclusion