Implement local OCR and pre-persistence redaction #51

Merged

lost-rob0t merged 1 commit from agent/issue-9-redaction into main

2026-07-18 17:13:54 +00:00

lost-rob0t commented

2026-07-18 14:46:13 +00:00

(Migrated from github.com)

What changed

Added a local tesseract-local OCR provider that streams in-memory P5/P6 frames over standard input and parses TSV from standard output.
Added finite OCR input, timeout, executable, language, and output validation without shell execution, network access, or temporary image files.
Added deterministic secret detection for common cloud/developer token formats, authorization headers, private-key blocks, credential assignments, credential-bearing connection strings, account keys, JWTs, email addresses, custom expressions, encoded variants, and measured high-entropy values.
Added conservative low-confidence handling that replaces OCR text and masks the full OCR region.
Added complete OCR-region bounds validation before detection so malformed OCR cannot bypass redaction.
Added pixel masking, OCR-text replacement, and sensitive metadata-field removal before a RedactedStageItem may be constructed.
Added exact, pattern-scoped allowlists with bounded values, configuration-time target validation, hidden effective configuration, and digest-only runtime audit decisions.
Added strict versioned analyzed/redacted stage codecs with record, generation, revision, and frame-identity checks.
Added OCR and redaction stage processors for the existing bounded Pykka/ZeroMQ pipeline.
Added sanitized OCR/redaction errors containing record identity and fixed codes only.
Added Tesseract to the Nix development environment and documented the plaintext/redaction boundary.

Security behavior

Raw pixels, OCR text, and unredacted metadata exist only through the analyzed in-memory stage.
The redacted stage contains masked pixels, scrubbed OCR text, scrubbed metadata, and non-secret decision descriptors only.
No network OCR service, temporary image, filesystem spool, shell command, pickle payload, or fallback plaintext path exists.
Deterministic filtering always runs before optional model assistance; model assistance cannot be the sole control or define allowlists.
Invalid OCR regions, stale policy revisions, identity mismatches, codec failures, detector failures, or redaction failures reject the complete record.
Failed redaction produces no encryption or storage write.
Logs and exceptions omit OCR output, matched values, metadata values, subprocess stderr, and pixel content.

TDD evidence

Observed red results:

The first OCR-region implementation validated bounds only after a detector matched. A synthetic out-of-frame OCR region with ordinary text therefore escaped rejection. Bounds validation was moved ahead of all detection, and malformed OCR now rejects the complete record unconditionally.
The initial synthetic Google-token fixture had the wrong provider length and correctly failed to match. The fixture was corrected to the documented provider shape.
Secret scanning rejected literal provider-shaped test tokens. Synthetic corpus values are now assembled at runtime without weakening detector coverage or adding broad scanner exclusions.
Pipeline integration initially depended on unit-test package imports. Integration fixtures were made package-local so CI collection does not depend on the top-level tests namespace.

Focused green result:

146 tests passed locally under the exact Python 3.14 environment.
Ruff formatting and linting passed.
ShellCheck passed.
Strict Pyright passed with zero errors.
Repository policy checks, Bandit, and detect-secrets passed.
Tests cover provider-shaped and encoded token variants, entropy true/false positives, exact allowlists, digest-only audit, Tesseract stdin/stdout operation, pixel/text/metadata redaction, low-confidence masking, strict codecs, pipeline persistence boundaries, complete-record rejection, and sanitized faults.

Python 3.14 CI validation

Final workflow run #68 passed:

Ubuntu 22.04: formatting, Ruff lint, ShellCheck, strict Pyright, unit tests, contract tests, integration tests, security tests, and security scans passed.
Ubuntu 24.04: formatting, Ruff lint, ShellCheck, strict Pyright, unit tests, contract tests, integration tests, security tests, and security scans passed.
Independent eight-mode failure propagation passed.

Acceptance criteria

OCR is local and does not use a remote service.
Sensitive pixels, OCR text, and metadata are redacted before encryption or persistence.
Low-confidence and malformed OCR fail closed.
Allowlists are narrow, validated, and auditable without storing allowed values.
Raw OCR and unredacted content do not appear in logs or storage.
Redaction failure rejects the whole record.

Closes #9

## What changed - Added a local `tesseract-local` OCR provider that streams in-memory P5/P6 frames over standard input and parses TSV from standard output. - Added finite OCR input, timeout, executable, language, and output validation without shell execution, network access, or temporary image files. - Added deterministic secret detection for common cloud/developer token formats, authorization headers, private-key blocks, credential assignments, credential-bearing connection strings, account keys, JWTs, email addresses, custom expressions, encoded variants, and measured high-entropy values. - Added conservative low-confidence handling that replaces OCR text and masks the full OCR region. - Added complete OCR-region bounds validation before detection so malformed OCR cannot bypass redaction. - Added pixel masking, OCR-text replacement, and sensitive metadata-field removal before a `RedactedStageItem` may be constructed. - Added exact, pattern-scoped allowlists with bounded values, configuration-time target validation, hidden effective configuration, and digest-only runtime audit decisions. - Added strict versioned analyzed/redacted stage codecs with record, generation, revision, and frame-identity checks. - Added OCR and redaction stage processors for the existing bounded Pykka/ZeroMQ pipeline. - Added sanitized OCR/redaction errors containing record identity and fixed codes only. - Added Tesseract to the Nix development environment and documented the plaintext/redaction boundary. ## Security behavior - Raw pixels, OCR text, and unredacted metadata exist only through the analyzed in-memory stage. - The redacted stage contains masked pixels, scrubbed OCR text, scrubbed metadata, and non-secret decision descriptors only. - No network OCR service, temporary image, filesystem spool, shell command, pickle payload, or fallback plaintext path exists. - Deterministic filtering always runs before optional model assistance; model assistance cannot be the sole control or define allowlists. - Invalid OCR regions, stale policy revisions, identity mismatches, codec failures, detector failures, or redaction failures reject the complete record. - Failed redaction produces no encryption or storage write. - Logs and exceptions omit OCR output, matched values, metadata values, subprocess stderr, and pixel content. ## TDD evidence **Observed red results:** - The first OCR-region implementation validated bounds only after a detector matched. A synthetic out-of-frame OCR region with ordinary text therefore escaped rejection. Bounds validation was moved ahead of all detection, and malformed OCR now rejects the complete record unconditionally. - The initial synthetic Google-token fixture had the wrong provider length and correctly failed to match. The fixture was corrected to the documented provider shape. - Secret scanning rejected literal provider-shaped test tokens. Synthetic corpus values are now assembled at runtime without weakening detector coverage or adding broad scanner exclusions. - Pipeline integration initially depended on unit-test package imports. Integration fixtures were made package-local so CI collection does not depend on the top-level `tests` namespace. **Focused green result:** - 146 tests passed locally under the exact Python 3.14 environment. - Ruff formatting and linting passed. - ShellCheck passed. - Strict Pyright passed with zero errors. - Repository policy checks, Bandit, and detect-secrets passed. - Tests cover provider-shaped and encoded token variants, entropy true/false positives, exact allowlists, digest-only audit, Tesseract stdin/stdout operation, pixel/text/metadata redaction, low-confidence masking, strict codecs, pipeline persistence boundaries, complete-record rejection, and sanitized faults. ## Python 3.14 CI validation Final workflow run #68 passed: - Ubuntu 22.04: formatting, Ruff lint, ShellCheck, strict Pyright, unit tests, contract tests, integration tests, security tests, and security scans passed. - Ubuntu 24.04: formatting, Ruff lint, ShellCheck, strict Pyright, unit tests, contract tests, integration tests, security tests, and security scans passed. - Independent eight-mode failure propagation passed. ## Acceptance criteria - OCR is local and does not use a remote service. - Sensitive pixels, OCR text, and metadata are redacted before encryption or persistence. - Low-confidence and malformed OCR fail closed. - Allowlists are narrow, validated, and auditable without storing allowed values. - Raw OCR and unredacted content do not appear in logs or storage. - Redaction failure rejects the whole record. Closes #9