Add idempotent batch ingest and target lifecycle APIs #59

New issue

Open

opened 2026-07-26 02:34:51 +00:00 by lost-rob0t · 2 comments

lost-rob0t commented

2026-07-26 02:34:51 +00:00

(Migrated from github.com)

Goal

Replace single-document fire-and-forget routes with schema-aware, idempotent ingest and a complete target lifecycle.

Scope

POST /api/v1/documents:batch with atomic/non-atomic modes, per-record validation, dedupe, replacement policy, and origin metadata
POST /api/v1/targets, GET /api/v1/targets, GET/PATCH/DELETE /api/v1/targets/:id
explicit target states: queued, scheduled, running, completed, failed, cancelled
actor registration/capability validation before scheduling
one-shot and recurring scheduling with durable recovery
idempotency keys and request correlation IDs
emit typed RabbitMQ events for accepted/rejected documents and target state transitions
preserve a compatibility adapter for /new/document/:dtype and /new/target/:actor

Acceptance criteria

v0.9 documents are validated before publish/persist
clients can safely retry without duplicating documents or target runs
batch results identify saved, skipped, invalid, and failed records
target state survives restart and can be queried/cancelled
wire keys and schema revision are documented and contract-tested

Depends on #58 for the versioned API envelope and capability discovery.

## Goal Replace single-document fire-and-forget routes with schema-aware, idempotent ingest and a complete target lifecycle. ## Scope - `POST /api/v1/documents:batch` with atomic/non-atomic modes, per-record validation, dedupe, replacement policy, and origin metadata - `POST /api/v1/targets`, `GET /api/v1/targets`, `GET/PATCH/DELETE /api/v1/targets/:id` - explicit target states: queued, scheduled, running, completed, failed, cancelled - actor registration/capability validation before scheduling - one-shot and recurring scheduling with durable recovery - idempotency keys and request correlation IDs - emit typed RabbitMQ events for accepted/rejected documents and target state transitions - preserve a compatibility adapter for `/new/document/:dtype` and `/new/target/:actor` ## Acceptance criteria - v0.9 documents are validated before publish/persist - clients can safely retry without duplicating documents or target runs - batch results identify saved, skipped, invalid, and failed records - target state survives restart and can be queried/cancelled - wire keys and schema revision are documented and contract-tested Depends on #58 for the versioned API envelope and capability discovery.

lost-rob0t commented

2026-08-23 03:07:45 +00:00

(Migrated from github.com)

Current production behavior was re-verified on 2026-08-22 while designing Auto-Dig operations skills. The internal service advertises doc_spec_version 0.8.0, while the Auto-Dig canonical repository schema is 0.9.0. The current legacy POST /new/document/:dtype path performs transport-level shape and dtype checks but does not invoke strict StarIntel 0.9 validation; HTTP success means RabbitMQ acceptance, not persistence. This confirms the schema-aware and idempotent v1 ingest acceptance criteria remain necessary.

lost-rob0t commented

2026-08-26 02:27:04 +00:00

(Migrated from github.com)

RAGE unblock slice started in draft PR #127 from immutable master 52948721d8981e9aa5ccad4efd77185e52af845b.

Architecture correction from current product decision: this issue owns the generic target lifecycle/idempotency API. Bixby-specific prepare/commit projection is an optional starintel-bixby add-on over this core API; there is no standalone Bixby gateway.

The smallest first slice is POST /api/v1/targets with targets:dispatch, principal-bound deterministic idempotency, narrow receipt output, and reuse of the existing durable acceptance/schedule machinery. Tests also pin a newly found owner defect: the current target acceptance fingerprint omits target value/options/recurrence, so changed content under one schedule identity can be mistaken for a duplicate. That must fail as conflict before Bixby target commit is enabled.

RAGE unblock slice started in draft PR #127 from immutable master `52948721d8981e9aa5ccad4efd77185e52af845b`. Architecture correction from current product decision: this issue owns the **generic** target lifecycle/idempotency API. Bixby-specific prepare/commit projection is an optional `starintel-bixby` add-on over this core API; there is no standalone Bixby gateway. The smallest first slice is `POST /api/v1/targets` with `targets:dispatch`, principal-bound deterministic idempotency, narrow receipt output, and reuse of the existing durable acceptance/schedule machinery. Tests also pin a newly found owner defect: the current target acceptance fingerprint omits target value/options/recurrence, so changed content under one schedule identity can be mistaken for a duplicate. That must fail as conflict before Bixby target commit is enabled.