[P1-08] Add structured metrics, tracing, audit events, and operational diagnostics #46

New issue

Open

opened 2026-07-22 00:19:42 +00:00 by lost-rob0t · 1 comment

lost-rob0t commented

2026-07-22 00:19:42 +00:00

(Migrated from github.com)

Problem

Current observability is primarily ad hoc log messages. There is no stable correlation across HTTP, Rabbit, actors, CouchDB, target schedules, leases, retries, and generated documents. Queue pressure and runtime ownership cannot be inspected reliably.

Required metrics

runtime-owned threads and thread-budget allocation
actor/component counts and generations
mailbox depth/capacity, processed, rejected, dropped, and handler latency
dispatcher workers, saturation, queue depth, and task latency
router membership, route selection, failures, and pressure
Rabbit connection/channel health, in-flight, unsettled, retries, DLQ, and publish latency
CouchDB pool usage, latency, conflicts, retries, and outbox backlog
active target schedules, execution attempts, lateness, failures, and recovery
target leases, conflicts, renew failures, expirations, revocations, and stale-fence rejects
HTTP request rate, latency, status, authentication failures, authorization denies, and rate limiting

Tracing and audit

Propagate trace id, parent span, request/message id, principal, component, target/job id, deadline, and attempt through every envelope.
Emit typed lifecycle, security, lease, overload, retry, persistence, and target-execution events.
Audit records must be tamper-evident or exported to an append-only sink and must redact secrets/sensitive payloads according to policy.
Provide a diagnostic snapshot API restricted to operators.

Acceptance tests

One end-to-end document/target flow can be reconstructed by trace id.
Metrics expose queue saturation before work is dropped.
Credential values and restricted document content do not appear in metrics/logs/audit fixtures.
Audit events identify principal, action, resource, outcome, and fencing generation where applicable.

ARDR reconciliation — 2026-08-29

Canonical research/design now owns this scope:

lost-rob0t/starintel-auto-research/roam/research/star-server/STAR-RESEARCH-052-observability-correlation-audit-contract.org
lost-rob0t/starintel-auto-research/roam/design/star-server/STAR-SERVER-051-observability-correlation-audit-contract.org
Auto-Research transaction: lost-rob0t/starintel-auto-research#201

Lifecycle only:

research: READY_FOR_DESIGN
design: DESIGN_READY_FOR_OPERATOR_REVIEW
implementation: PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL

No issue text, label, green test, publication state, or AUTO-RAGE wording is implementation authorization. Do not enqueue this issue into executable RAGE until the canonical operator approval record explicitly authorizes implementation.

Scope reconciliation

This issue is the parent semantic/diagnostic contract. Related open seeds are not independent competing architectures:

#121 is the structured-log/event projection over the canonical typed event model.
#122 remains the remote syslog transport adapter.
#128 is the optional generic OTLP/OpenObserve exporter over canonical events/metrics/spans.
lost-rob0t/star-lang#49 owns StarLang actor/runtime instrumentation semantics.

OTLP/OpenObserve is a downstream optional adapter, not the internal application model. StarLang remains authoritative for actor/domain lifecycle, mailbox, supervision, dispatcher, and runtime-event semantics.

Architecture constraints

Extend/compose the existing explicit service/request context; do not replace it with an ambient telemetry global.
W3C traceparent/tracestate parsing lives at transport adapters; internal propagation uses validated typed context.
Operational events, metrics, spans, and durable audit records are distinct typed semantics with different cardinality/failure/durability rules.
Metric labels are bounded-cardinality; target/principal/document/request/trace IDs are not metric dimensions.
Forbidden secret/sensitive fields are rejected before sink invocation, not merely redacted after serialization.
Best-effort observability sink/export failure cannot change normal application results.
Durable audit has its own explicit append/commit/outcome contract; any operation that must fail closed on audit failure requires separate operator-approved policy.
Diagnostics aggregate typed read models from existing owners; HTTP must not scrape process globals directly.
#128 configuration must use the canonical data-only deployment configuration + secret-reference boundary from the deployment-config design. Executable init Lisp is trusted extension code, not routine telemetry configuration.
Default-disabled OTLP means zero exporter creation, outbound network, export worker/thread, or secret-resolution side effects.

Proposed implementation slices — not approved

Pure semantic values + allowlist validation (starintel-server). No HTTP/actor/Rabbit/CouchDB/network/thread effects.
Explicit HTTP/service W3C trace propagation while preserving existing correlation IDs.
One authoritative bounded metric vertical at an actual queue/admission owner.
Structured event/log projection under #121.
StarLang actor/runtime instrumentation under star-lang#49.
Durable audit-sink protocol + fake conformance suite before selecting a concrete durable sink.
Optional OTLP/OpenObserve adapter under #128 using the closed config/secret-ref boundary.
Operator diagnostic snapshot composed from typed runtime/component read ports.

Mandatory RED-first target for slice 1

On untouched master, define a test against the new typed constructor:

ctx = ObservationContext(correlationId="c-1",
                         traceId="4bf92f3577b34da6a3ce929d0e0e4736",
                         spanId="00f067aa0ba902b7")

makeOperationalEvent(
  name="security.authentication.denied",
  context=ctx,
  attributes={action="authenticate",
              authorization="Bearer super-secret"})

=> forbidden-observation-field("authorization")
=> fake sink call count = 0
=> no serialized/rendered output contains "super-secret"

A post-hoc regex redaction test over a free-form log string is not valid GREEN.

Sibling REDs: unknown attributes; oversize values; invalid trace/span IDs; forbidden high-cardinality metric labels; deterministic schema/version; immutable/copy-safe context; disabled OTLP creates zero network/thread/exporter effects.

Dependencies

Pure semantic values can be implemented independently after operator approval. Production propagation/snapshot composition should consume the #39/#40/#42 explicit runtime/component seams as realized. Queue/lifecycle metrics consume #34–#37/#42 ownership. OTLP config consumes #38. Actor/runtime instrumentation is owned by StarLang #49.

These dependencies affect realization order only; none of them grant approval.

## Problem Current observability is primarily ad hoc log messages. There is no stable correlation across HTTP, Rabbit, actors, CouchDB, target schedules, leases, retries, and generated documents. Queue pressure and runtime ownership cannot be inspected reliably. ## Required metrics - runtime-owned threads and thread-budget allocation - actor/component counts and generations - mailbox depth/capacity, processed, rejected, dropped, and handler latency - dispatcher workers, saturation, queue depth, and task latency - router membership, route selection, failures, and pressure - Rabbit connection/channel health, in-flight, unsettled, retries, DLQ, and publish latency - CouchDB pool usage, latency, conflicts, retries, and outbox backlog - active target schedules, execution attempts, lateness, failures, and recovery - target leases, conflicts, renew failures, expirations, revocations, and stale-fence rejects - HTTP request rate, latency, status, authentication failures, authorization denies, and rate limiting ## Tracing and audit - Propagate trace id, parent span, request/message id, principal, component, target/job id, deadline, and attempt through every envelope. - Emit typed lifecycle, security, lease, overload, retry, persistence, and target-execution events. - Audit records must be tamper-evident or exported to an append-only sink and must redact secrets/sensitive payloads according to policy. - Provide a diagnostic snapshot API restricted to operators. ## Acceptance tests - One end-to-end document/target flow can be reconstructed by trace id. - Metrics expose queue saturation before work is dropped. - Credential values and restricted document content do not appear in metrics/logs/audit fixtures. - Audit events identify principal, action, resource, outcome, and fencing generation where applicable. --- ## ARDR reconciliation — 2026-08-29 Canonical research/design now owns this scope: - `lost-rob0t/starintel-auto-research/roam/research/star-server/STAR-RESEARCH-052-observability-correlation-audit-contract.org` - `lost-rob0t/starintel-auto-research/roam/design/star-server/STAR-SERVER-051-observability-correlation-audit-contract.org` - Auto-Research transaction: `lost-rob0t/starintel-auto-research#201` Lifecycle only: - research: `READY_FOR_DESIGN` - design: `DESIGN_READY_FOR_OPERATOR_REVIEW` - implementation: **PENDING / AWAITING_OPERATOR_IMPLEMENTATION_APPROVAL** No issue text, label, green test, publication state, or AUTO-RAGE wording is implementation authorization. Do not enqueue this issue into executable RAGE until the canonical operator approval record explicitly authorizes implementation. ### Scope reconciliation This issue is the parent semantic/diagnostic contract. Related open seeds are not independent competing architectures: - #121 is the structured-log/event projection over the canonical typed event model. - #122 remains the remote syslog transport adapter. - #128 is the optional generic OTLP/OpenObserve exporter over canonical events/metrics/spans. - `lost-rob0t/star-lang#49` owns StarLang actor/runtime instrumentation semantics. OTLP/OpenObserve is a downstream optional adapter, not the internal application model. StarLang remains authoritative for actor/domain lifecycle, mailbox, supervision, dispatcher, and runtime-event semantics. ### Architecture constraints - Extend/compose the existing explicit service/request context; do not replace it with an ambient telemetry global. - W3C `traceparent`/`tracestate` parsing lives at transport adapters; internal propagation uses validated typed context. - Operational events, metrics, spans, and durable audit records are distinct typed semantics with different cardinality/failure/durability rules. - Metric labels are bounded-cardinality; target/principal/document/request/trace IDs are not metric dimensions. - Forbidden secret/sensitive fields are rejected before sink invocation, not merely redacted after serialization. - Best-effort observability sink/export failure cannot change normal application results. - Durable audit has its own explicit append/commit/outcome contract; any operation that must fail closed on audit failure requires separate operator-approved policy. - Diagnostics aggregate typed read models from existing owners; HTTP must not scrape process globals directly. - #128 configuration must use the canonical data-only deployment configuration + secret-reference boundary from the deployment-config design. Executable init Lisp is trusted extension code, not routine telemetry configuration. - Default-disabled OTLP means zero exporter creation, outbound network, export worker/thread, or secret-resolution side effects. ### Proposed implementation slices — not approved 1. **Pure semantic values + allowlist validation** (`starintel-server`). No HTTP/actor/Rabbit/CouchDB/network/thread effects. 2. **Explicit HTTP/service W3C trace propagation** while preserving existing correlation IDs. 3. **One authoritative bounded metric vertical** at an actual queue/admission owner. 4. **Structured event/log projection** under #121. 5. **StarLang actor/runtime instrumentation** under `star-lang#49`. 6. **Durable audit-sink protocol + fake conformance suite** before selecting a concrete durable sink. 7. **Optional OTLP/OpenObserve adapter** under #128 using the closed config/secret-ref boundary. 8. **Operator diagnostic snapshot** composed from typed runtime/component read ports. ### Mandatory RED-first target for slice 1 On untouched `master`, define a test against the new typed constructor: ```text ctx = ObservationContext(correlationId="c-1", traceId="4bf92f3577b34da6a3ce929d0e0e4736", spanId="00f067aa0ba902b7") makeOperationalEvent( name="security.authentication.denied", context=ctx, attributes={action="authenticate", authorization="Bearer super-secret"}) => forbidden-observation-field("authorization") => fake sink call count = 0 => no serialized/rendered output contains "super-secret" ``` A post-hoc regex redaction test over a free-form log string is **not** valid GREEN. Sibling REDs: unknown attributes; oversize values; invalid trace/span IDs; forbidden high-cardinality metric labels; deterministic schema/version; immutable/copy-safe context; disabled OTLP creates zero network/thread/exporter effects. ### Dependencies Pure semantic values can be implemented independently **after operator approval**. Production propagation/snapshot composition should consume the #39/#40/#42 explicit runtime/component seams as realized. Queue/lifecycle metrics consume #34–#37/#42 ownership. OTLP config consumes #38. Actor/runtime instrumentation is owned by StarLang #49. These dependencies affect realization order only; none of them grant approval.

nsaspy referenced this issue

2026-08-28 03:29:41 +00:00

Lock server to canonical research-node schema #78

nsaspy commented

2026-09-02 17:26:53 +00:00

Owner

Operator approval recorded for STAR-RESEARCH-052 — Observability Correlation, Metrics, and Audit Contract.

Approved research evidence: github.com/lost-rob0t/starintel-auto-research@0b5ac74a36

Scope of this approval: research conclusion approved; proceed to canonical design. This does not approve the architecture/design itself and does not authorize implementation. Those remain separate operator gates.

Operator approval recorded for **STAR-RESEARCH-052 — Observability Correlation, Metrics, and Audit Contract**. Approved research evidence: https://github.com/lost-rob0t/starintel-auto-research/commit/0b5ac74a36c5b6500df7fb494253ed225a0ab2b5 Scope of this approval: **research conclusion approved; proceed to canonical design**. This does **not** approve the architecture/design itself and does **not** authorize implementation. Those remain separate operator gates.

nsaspy referenced this issue

2026-09-04 14:37:38 +00:00

[P1-09] Add end-to-end idempotency and duplicate suppression across HTTP, Rabbit, persistence, targets, and leases #47