Build bounded in-memory ZeroMQ capture pipeline #50

Merged

lost-rob0t merged 20 commits from agent/issue-8-pipeline into main

2026-07-18 14:44:10 +00:00

lost-rob0t commented

2026-07-18 07:41:49 +00:00

(Migrated from github.com)

What changed

Added a four-stage typed pipeline: raw, analyzed, redacted, encrypted.
Added one bounded Pykka ThreadingActor pump per stage.
Added a single process-owned PyZMQ context with fixed-role inproc:// PUSH/PULL edges only.
Added single-thread socket ownership, explicit HWM, finite send/receive deadlines, IMMEDIATE=1, and LINGER=0.
Added strict Pydantic multipart framing with protocol version, record ID, generation, configuration revision, deadline, frame count, and exact frame sizes.
Added generation-scoped application credit ledgers as the authoritative bound beyond ZeroMQ HWM.
Added drop-newest and one-item coalesce-latest raw overload policies.
Added mutable raw-buffer zeroing after copy, drop, replacement, drain, or rejection.
Added lifecycle cancellation propagation and generation checks before processing, forwarding, and persistence.
Added final encrypted persistence through CaptureGate.run_persistence().
Added sanitized stage faults and a lifecycle actor-fault bridge.
Added thread-safe out-of-band stage metrics so status remains observable while processors are blocked.
Added validated capture configuration for raw queue size, later-stage queue size, and overload policy.
Added required integration-test execution to CI.

Security behavior

Raw and intermediate stage data use inproc:// only.
No temporary image files, filesystem spool, durable broker, pickle, or secondary fallback queue exists.
Each edge has a finite HWM and a stricter finite application credit limit.
Every item carries the authoritative capture generation and immutable configuration revision.
Stop/fault cancellation destroys coalesced work, rejects stale queued work, waits for generation credits to drain, and prevents stale persistence.
Worker and protocol errors expose record UUID, stage, and fixed fault code only; exception text and binary content are discarded.
Socket shutdown uses zero linger and the Pykka registry must be clean after pipeline close.

TDD evidence

Observed red results:

Initial focused tests failed because the pipeline package and transport stages did not exist.
A regression test exposed that querying status through Pykka ask() blocked behind a deliberately stalled processor. Metrics were moved to a thread-safe shared snapshot, preserving observability without draining payloads into actor inboxes.
Strict Pyright exposed weak boundaries around JSON dictionaries, PyZMQ multipart methods, Pykka return unions, and lifecycle permits. Those interfaces were explicitly typed rather than suppressed.
The clean CI lint gate exposed repository-specific import classification. Ruff's pinned canonical diff was applied exactly; diagnostic workflow changes were then removed.
The first integration run failed during collection because it imported fixtures through the top-level tests.unit package. Integration fixtures now live beside the integration tests and use a relative import, eliminating cross-suite import assumptions.

Focused green result:

57 issue-focused tests passed during the local development loop.
Ruff formatting and linting passed against the Python 3.14 target.
Strict Pyright passed with zero errors.
Tests cover framing validation, socket ownership/options, hard credit bounds, drop/coalesce behavior, raw-buffer zeroing, no filesystem backing, stop cancellation, stale persistence rejection, sanitized faults, lifecycle fault bridging, configuration mapping, and actor-registry cleanup.

Python 3.14 CI validation

Final workflow run #58 passed:

Ubuntu 22.04: formatting, Ruff lint, ShellCheck, strict Pyright, unit tests, contract tests, integration tests, security tests, and security scans passed.
Ubuntu 24.04: formatting, Ruff lint, ShellCheck, strict Pyright, unit tests, contract tests, integration tests, security tests, and security scans passed.
Independent eight-mode failure propagation passed.

Acceptance criteria

Typed stages are bounded by HWM plus application credits.
Raw buffers remain process-local and are destroyed after use.
Disabling capture drains or destroys pending work without storage writes.
Overload drops or coalesces within fixed bounds.
Fault messages contain record IDs and fixed codes, never captured content.
Integration tests run as a required CI layer.

Closes #8

## What changed - Added a four-stage typed pipeline: raw, analyzed, redacted, encrypted. - Added one bounded Pykka `ThreadingActor` pump per stage. - Added a single process-owned PyZMQ context with fixed-role `inproc://` PUSH/PULL edges only. - Added single-thread socket ownership, explicit HWM, finite send/receive deadlines, `IMMEDIATE=1`, and `LINGER=0`. - Added strict Pydantic multipart framing with protocol version, record ID, generation, configuration revision, deadline, frame count, and exact frame sizes. - Added generation-scoped application credit ledgers as the authoritative bound beyond ZeroMQ HWM. - Added `drop-newest` and one-item `coalesce-latest` raw overload policies. - Added mutable raw-buffer zeroing after copy, drop, replacement, drain, or rejection. - Added lifecycle cancellation propagation and generation checks before processing, forwarding, and persistence. - Added final encrypted persistence through `CaptureGate.run_persistence()`. - Added sanitized stage faults and a lifecycle actor-fault bridge. - Added thread-safe out-of-band stage metrics so status remains observable while processors are blocked. - Added validated capture configuration for raw queue size, later-stage queue size, and overload policy. - Added required integration-test execution to CI. ## Security behavior - Raw and intermediate stage data use `inproc://` only. - No temporary image files, filesystem spool, durable broker, pickle, or secondary fallback queue exists. - Each edge has a finite HWM and a stricter finite application credit limit. - Every item carries the authoritative capture generation and immutable configuration revision. - Stop/fault cancellation destroys coalesced work, rejects stale queued work, waits for generation credits to drain, and prevents stale persistence. - Worker and protocol errors expose record UUID, stage, and fixed fault code only; exception text and binary content are discarded. - Socket shutdown uses zero linger and the Pykka registry must be clean after pipeline close. ## TDD evidence **Observed red results:** - Initial focused tests failed because the pipeline package and transport stages did not exist. - A regression test exposed that querying status through Pykka `ask()` blocked behind a deliberately stalled processor. Metrics were moved to a thread-safe shared snapshot, preserving observability without draining payloads into actor inboxes. - Strict Pyright exposed weak boundaries around JSON dictionaries, PyZMQ multipart methods, Pykka return unions, and lifecycle permits. Those interfaces were explicitly typed rather than suppressed. - The clean CI lint gate exposed repository-specific import classification. Ruff's pinned canonical diff was applied exactly; diagnostic workflow changes were then removed. - The first integration run failed during collection because it imported fixtures through the top-level `tests.unit` package. Integration fixtures now live beside the integration tests and use a relative import, eliminating cross-suite import assumptions. **Focused green result:** - 57 issue-focused tests passed during the local development loop. - Ruff formatting and linting passed against the Python 3.14 target. - Strict Pyright passed with zero errors. - Tests cover framing validation, socket ownership/options, hard credit bounds, drop/coalesce behavior, raw-buffer zeroing, no filesystem backing, stop cancellation, stale persistence rejection, sanitized faults, lifecycle fault bridging, configuration mapping, and actor-registry cleanup. ## Python 3.14 CI validation Final workflow run #58 passed: - Ubuntu 22.04: formatting, Ruff lint, ShellCheck, strict Pyright, unit tests, contract tests, integration tests, security tests, and security scans passed. - Ubuntu 24.04: formatting, Ruff lint, ShellCheck, strict Pyright, unit tests, contract tests, integration tests, security tests, and security scans passed. - Independent eight-mode failure propagation passed. ## Acceptance criteria - Typed stages are bounded by HWM plus application credits. - Raw buffers remain process-local and are destroyed after use. - Disabling capture drains or destroys pending work without storage writes. - Overload drops or coalesces within fixed bounds. - Fault messages contain record IDs and fixed codes, never captured content. - Integration tests run as a required CI layer. Closes #8