EPIC: native StarIntel webhook subscriptions, durable delivery and event registry #129

New issue

Open

opened 2026-08-26 08:38:40 +00:00 by lost-rob0t · 1 comment

lost-rob0t commented

2026-08-26 08:38:40 +00:00

(Migrated from github.com)

Goal

Make webhook/event support a first-class native StarIntel server capability so external gateways, internal services and automation consume one canonical event model instead of inventing parallel systems.

Parent integration registry: #124.
Infra/public gateway: starintel-labs/starintel-infra#64 (webhooks.starintel.actor).

Architecture

starintel-server owns canonical event semantics, subscriptions and durable delivery state.

The public webhooks.starintel.actor gateway handles Internet/provider ingress, user-facing configuration and external-adapter concerns, then translates into/from this canonical server contract where appropriate.

Node-RED is a downstream operator automation consumer, never the webhook authority.

Required model

Define versioned types/contracts for:

event type registry
stable event IDs
event schema/version
event occurrence timestamp
subject/actor identity where applicable
provenance/source
subscription identity
subscription enable/disable state
destination/integration reference without exposing arbitrary payload-controlled URLs
signing/secret version
delivery attempt state/history
next retry/dead-letter state
idempotency/replay metadata

Outbound webhook delivery

durable outbox: state mutation and event publication must not silently diverge
deterministic serialized bytes for signing and exact-byte/hash validation
HMAC/signature scheme with versioned headers
timestamp/freshness semantics
bounded connection/request/response size and timeout
exponential backoff with capped attempts/jitter
permanent vs retryable error classification
dead-letter state
per-subscription pause/disable
secret rotation without losing delivery verification continuity
delivery history/query surface
duplicate/restart-safe behavior

Inbound canonical events

Provide a narrow authenticated API/adapter contract for approved normalized events from webhooks.starintel.actor and internal integrations.

Requirements:

caller/service authentication and authorization
stable external/provider event identity
replay/idempotency protection
bounded input/schema validation
explicit source/provider provenance
no payload-controlled shell commands, RabbitMQ routing keys, arbitrary internal URLs, Node-RED flow IDs or management actions
event acceptance must not itself imply authorization for downstream mutations

Authorization

Separate:

event visibility
subscription management
delivery administration/replay
event production
downstream action authority

A normal StarIntel user account must not gain operator/admin event controls merely by existing.

Integration registry

Extend/reuse #124 rather than creating another service-discovery/config mechanism. Runtime destinations/credentials are resolved through registered integrations/credential references and never leaked through public capabilities/OpenAPI.

Observability

Emit OTLP logs/metrics/traces/audit events compatible with the StarIntel OpenObserve plane:

event count by type
enqueue/publish/delivery latency
attempts/retries/failures
dead-letter count
subscription state changes
denied/invalid/replayed event counts

Never log webhook secrets, Authorization headers, tokens or sensitive raw payloads by default.

TDD / threat model

Tests first for:

transaction commits but publisher crashes before delivery -> event remains recoverable
duplicate producer submission -> one canonical event outcome
restart during delivery -> no uncontrolled duplicate mutation
forged/invalid signatures
replay/stale timestamps
secret rotation
oversized/malformed payloads
redirect/SSRF attempts
destination allowlist/integration reference enforcement
confused-deputy attempts to turn an event into unauthorized control action
dead-letter/replay authorization
exact serialized byte/hash/signature validation
telemetry redaction

Acceptance

A canonical StarIntel mutation atomically/durably produces an event/outbox record.
Approved subscription receives the deterministic signed event.
Retry/restart behavior is idempotent and bounded.
Failed deliveries reach inspectable dead-letter state.
Approved normalized external event from the infra gateway is accepted exactly once.
Forged/replayed/unauthorized events fail closed.
Delivery/subscription controls enforce user/operator authorization boundaries.
OpenObserve-compatible telemetry is emitted without secrets.
Public capability discovery advertises webhook/event capability without leaking private topology.
Repo-native tests and exact-head CI are green.

Execution mode — AUTO-RAGE

Run the full ADARD/ADADR loop autonomously. Inspect current server event/outbox/idempotency primitives and #124 first; adversarial design + threat model; TDD-first implementation in dependency slices; exact-head service-backed verification; focused PR(s) targeting master. Stop only for a genuine unresolved requirement/security blocker, inaccessible dependency or failed verification gate.

## Goal Make webhook/event support a first-class **native StarIntel server capability** so external gateways, internal services and automation consume one canonical event model instead of inventing parallel systems. Parent integration registry: #124. Infra/public gateway: `starintel-labs/starintel-infra#64` (`webhooks.starintel.actor`). ## Architecture `starintel-server` owns canonical event semantics, subscriptions and durable delivery state. The public `webhooks.starintel.actor` gateway handles Internet/provider ingress, user-facing configuration and external-adapter concerns, then translates into/from this canonical server contract where appropriate. Node-RED is a downstream operator automation consumer, never the webhook authority. ## Required model Define versioned types/contracts for: - event type registry - stable event IDs - event schema/version - event occurrence timestamp - subject/actor identity where applicable - provenance/source - subscription identity - subscription enable/disable state - destination/integration reference without exposing arbitrary payload-controlled URLs - signing/secret version - delivery attempt state/history - next retry/dead-letter state - idempotency/replay metadata ## Outbound webhook delivery - durable outbox: state mutation and event publication must not silently diverge - deterministic serialized bytes for signing and exact-byte/hash validation - HMAC/signature scheme with versioned headers - timestamp/freshness semantics - bounded connection/request/response size and timeout - exponential backoff with capped attempts/jitter - permanent vs retryable error classification - dead-letter state - per-subscription pause/disable - secret rotation without losing delivery verification continuity - delivery history/query surface - duplicate/restart-safe behavior ## Inbound canonical events Provide a narrow authenticated API/adapter contract for approved normalized events from `webhooks.starintel.actor` and internal integrations. Requirements: - caller/service authentication and authorization - stable external/provider event identity - replay/idempotency protection - bounded input/schema validation - explicit source/provider provenance - no payload-controlled shell commands, RabbitMQ routing keys, arbitrary internal URLs, Node-RED flow IDs or management actions - event acceptance must not itself imply authorization for downstream mutations ## Authorization Separate: - event visibility - subscription management - delivery administration/replay - event production - downstream action authority A normal StarIntel user account must not gain operator/admin event controls merely by existing. ## Integration registry Extend/reuse #124 rather than creating another service-discovery/config mechanism. Runtime destinations/credentials are resolved through registered integrations/credential references and never leaked through public capabilities/OpenAPI. ## Observability Emit OTLP logs/metrics/traces/audit events compatible with the StarIntel OpenObserve plane: - event count by type - enqueue/publish/delivery latency - attempts/retries/failures - dead-letter count - subscription state changes - denied/invalid/replayed event counts Never log webhook secrets, Authorization headers, tokens or sensitive raw payloads by default. ## TDD / threat model Tests first for: - transaction commits but publisher crashes before delivery -> event remains recoverable - duplicate producer submission -> one canonical event outcome - restart during delivery -> no uncontrolled duplicate mutation - forged/invalid signatures - replay/stale timestamps - secret rotation - oversized/malformed payloads - redirect/SSRF attempts - destination allowlist/integration reference enforcement - confused-deputy attempts to turn an event into unauthorized control action - dead-letter/replay authorization - exact serialized byte/hash/signature validation - telemetry redaction ## Acceptance 1. A canonical StarIntel mutation atomically/durably produces an event/outbox record. 2. Approved subscription receives the deterministic signed event. 3. Retry/restart behavior is idempotent and bounded. 4. Failed deliveries reach inspectable dead-letter state. 5. Approved normalized external event from the infra gateway is accepted exactly once. 6. Forged/replayed/unauthorized events fail closed. 7. Delivery/subscription controls enforce user/operator authorization boundaries. 8. OpenObserve-compatible telemetry is emitted without secrets. 9. Public capability discovery advertises webhook/event capability without leaking private topology. 10. Repo-native tests and exact-head CI are green. ## Execution mode — AUTO-RAGE Run the full ADARD/ADADR loop autonomously. Inspect current server event/outbox/idempotency primitives and #124 first; adversarial design + threat model; TDD-first implementation in dependency slices; exact-head service-backed verification; focused PR(s) targeting `master`. Stop only for a genuine unresolved requirement/security blocker, inaccessible dependency or failed verification gate.

lost-rob0t commented

2026-08-26 13:11:20 +00:00

(Migrated from github.com)

RAGE/ADADR governance reconciliation: this is a new architecture scope, so the AUTO-RAGE execution section in this server issue does not authorize autonomous design/realization under the current StarIntel control-plane rules.

I opened the authoritative research-only transaction as lost-rob0t/starintel-auto-research#151 / STAR-RESEARCH-060:
https://github.com/lost-rob0t/starintel-auto-research/issues/151

Current gate:

research: NOT STARTED
operator research approval: PENDING
design/adversarial review: BLOCKED until explicit operator research approval
implementation: BLOCKED until a later canonical design receives explicit operator design approval

Preserve this server issue as requirements/input evidence. Do not treat its architecture prose, future green CI, or the existence of #151 as approval.

The research transaction also pins the hard datastore split: tek9 remains graph/KB-only (Prolog Actor -> Graph/KB boundary -> tek9), while CouchDB-backed document/state querying remains Prolog query server -> CouchDB. Webhook/event work must not collapse those planes.

No server code or tests should be created for #129 until the human-gated chain advances.

RAGE/ADADR governance reconciliation: this is a **new architecture scope**, so the `AUTO-RAGE` execution section in this server issue does not authorize autonomous design/realization under the current StarIntel control-plane rules. I opened the authoritative research-only transaction as **lost-rob0t/starintel-auto-research#151 / STAR-RESEARCH-060**: https://github.com/lost-rob0t/starintel-auto-research/issues/151 Current gate: - research: NOT STARTED - operator research approval: PENDING - design/adversarial review: BLOCKED until explicit operator research approval - implementation: BLOCKED until a later canonical design receives explicit operator design approval Preserve this server issue as requirements/input evidence. Do not treat its architecture prose, future green CI, or the existence of #151 as approval. The research transaction also pins the hard datastore split: tek9 remains graph/KB-only (`Prolog Actor -> Graph/KB boundary -> tek9`), while CouchDB-backed document/state querying remains `Prolog query server -> CouchDB`. Webhook/event work must not collapse those planes. No server code or tests should be created for #129 until the human-gated chain advances.