fix: emit v0.9-valid data.*-shaped target documents from both routes #144

Open

nsaspy wants to merge 0 commits from fix/target-dispatch-data-shape into master

nsaspy commented

2026-09-08 01:54:08 +00:00

Owner

Problem

Both live-server target dispatch routes emitted documents with TOP-LEVEL
actor/target/delay/recurring/options (and the v1 route,
schedule_id). The v0.9 schema's top level is closed
(additionalProperties: false) and has no such properties, so strict
consumers categorize these dispatches as parse errors. Verified against
staging bbpd: every live-route dispatch was answered with
event.parse-error and no tool ran, while a directly-published
data-shaped target round-tripped fully (fake tool executed, domains +
relations persisted in CouchDB, queue drained). The HTTP routes were the
last server-side emitter of the pre-0.9 shape.

Changes

/api/v1/targets (target-v1-document-from-request): target
semantics move into data; schedule identity moves to
extensions.schedule_id; the schema-required sources/evidence
collections are emitted.
/new/target/:actor: the legacy adapter folds its historical
top-level envelope into data (creating it when absent) and defaults
missing envelope fields (normalize-legacy-target-document); the URL
actor stays authoritative and caller-owned data values win over
folded top-level fields.
target-record-schedule-id reads extensions.schedule_id
(keeping v1 acceptance identities stable: target-request:<identity>);
pre-fix persisted documents keep working through the data/top-level
fallbacks, so recovery is unaffected.
publish-target-document-unchecked reads the actor through the
data-first accessor.
Strictness gap closed (side-finding from the dispatch
investigation): validate-document-input is now unconditionally
strict (the :strict-schema-p bypass is removed) and the target
compatibility consumer (handle-target) validates like every other
dtype. Previously top-level-shaped target documents entered the
pipeline unvalidated through both the legacy route and the
documents.new.target.<actor> consumer. The only remaining non-strict
decode is transport-metadata inspection (transient-p).

Local gates (hosted Actions distrusted; local evidence decides)

python3 -m unittest discover -s tests -p 'test_*.py' -v: 18/18 OK
nix run .#star-unit-tests: all suites green — http-boundary
32/32, http-target-v1 9/9, v09-runtime 8/8, plus all other required
suites (new tests assert the data shape, extensions schedule identity,
strict validation of the built v1 document, the legacy envelope fold,
caller-data precedence, rejection of undeclared legacy fields, strict
target delivery, and legacy persisted-document compatibility).
nix run .#star-integration-tests: blocked by a pre-existing,
load-sensitive flake in
one-hundred-concurrent-acquires-have-exactly-one-observed-owner:
under machine load (~11/32 cores) 1-2 of the 100 concurrent acquires
return :OUTCOME-UNKNOWN (the lease store's documented at-least-once
path: mutation submitted, response read lost; retry with the same
request-id). Identical failure was machine-recorded on clean master
c06b8fc (stash → gate → pop), so it pre-dates this change; this diff
touches no lease code. The runner aborts at that suite, so the
couchdb/http-api integration suites stay blocked behind it. Merging on
green python + unit gates per operator decision; the flake deserves a
separate test-robustness follow-up (retry retryable outcomes with the
same request-id per the documented contract).

Verification beyond gates

The fixed tree will be deployed to the live star-server and proven
end-to-end through the real HTTP path: .invalid targets POSTed via
both routes → acceptance docs → dispatch on
documents.target.dispatch.subfinder → staging bbpd executes the fake
tool → derived domains + relations persist in CouchDB under the staging
dataset → staging queue drains with no parse-error events; the malformed
fail-safe probe still holds.

## Problem Both live-server target dispatch routes emitted documents with TOP-LEVEL `actor`/`target`/`delay`/`recurring`/`options` (and the v1 route, `schedule_id`). The v0.9 schema's top level is closed (`additionalProperties: false`) and has no such properties, so strict consumers categorize these dispatches as parse errors. Verified against staging bbpd: every live-route dispatch was answered with `event.parse-error` and no tool ran, while a directly-published data-shaped target round-tripped fully (fake tool executed, domains + relations persisted in CouchDB, queue drained). The HTTP routes were the last server-side emitter of the pre-0.9 shape. ## Changes - **`/api/v1/targets`** (`target-v1-document-from-request`): target semantics move into `data`; schedule identity moves to `extensions.schedule_id`; the schema-required `sources`/`evidence` collections are emitted. - **`/new/target/:actor`**: the legacy adapter folds its historical top-level envelope into `data` (creating it when absent) and defaults missing envelope fields (`normalize-legacy-target-document`); the URL actor stays authoritative and caller-owned `data` values win over folded top-level fields. - **`target-record-schedule-id`** reads `extensions.schedule_id` (keeping v1 acceptance identities stable: `target-request:<identity>`); pre-fix persisted documents keep working through the data/top-level fallbacks, so recovery is unaffected. - **`publish-target-document-unchecked`** reads the actor through the data-first accessor. - **Strictness gap closed** (side-finding from the dispatch investigation): `validate-document-input` is now unconditionally strict (the `:strict-schema-p` bypass is removed) and the target compatibility consumer (`handle-target`) validates like every other dtype. Previously top-level-shaped target documents entered the pipeline unvalidated through both the legacy route and the `documents.new.target.<actor>` consumer. The only remaining non-strict decode is transport-metadata inspection (`transient-p`). ## Local gates (hosted Actions distrusted; local evidence decides) - `python3 -m unittest discover -s tests -p 'test_*.py' -v`: **18/18 OK** - `nix run .#star-unit-tests`: **all suites green** — http-boundary 32/32, http-target-v1 9/9, v09-runtime 8/8, plus all other required suites (new tests assert the data shape, extensions schedule identity, strict validation of the built v1 document, the legacy envelope fold, caller-data precedence, rejection of undeclared legacy fields, strict target delivery, and legacy persisted-document compatibility). - `nix run .#star-integration-tests`: **blocked by a pre-existing, load-sensitive flake** in `one-hundred-concurrent-acquires-have-exactly-one-observed-owner`: under machine load (~11/32 cores) 1-2 of the 100 concurrent acquires return `:OUTCOME-UNKNOWN` (the lease store's documented at-least-once path: mutation submitted, response read lost; retry with the same request-id). Identical failure was machine-recorded on **clean master c06b8fc** (stash → gate → pop), so it pre-dates this change; this diff touches no lease code. The runner aborts at that suite, so the couchdb/http-api integration suites stay blocked behind it. Merging on green python + unit gates per operator decision; the flake deserves a separate test-robustness follow-up (retry retryable outcomes with the same request-id per the documented contract). ## Verification beyond gates The fixed tree will be deployed to the live star-server and proven end-to-end through the real HTTP path: `.invalid` targets POSTed via both routes → acceptance docs → dispatch on `documents.target.dispatch.subfinder` → staging bbpd executes the fake tool → derived domains + relations persist in CouchDB under the staging dataset → staging queue drains with no parse-error events; the malformed fail-safe probe still holds.

Docstring coverage / doc-coverage (pull_request) Failing after 1m6s

Details

This branch is already included in the target branch. There is nothing to merge.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin fix/target-dispatch-data-shape:fix/target-dispatch-data-shape

git switch fix/target-dispatch-data-shape

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch master

git merge --no-ff fix/target-dispatch-data-shape

git switch fix/target-dispatch-data-shape

git rebase master

git switch master

git merge --ff-only fix/target-dispatch-data-shape

git switch fix/target-dispatch-data-shape

git rebase master

git switch master

git merge --no-ff fix/target-dispatch-data-shape

git switch master

git merge --squash fix/target-dispatch-data-shape