[P2] Add security regression, fuzz, and privacy-invariant tests #38

New issue

Closed

opened 2026-07-18 00:04:12 +00:00 by lost-rob0t · 1 comment

lost-rob0t commented

2026-07-18 00:04:12 +00:00

(Migrated from github.com)

Goal

Continuously prove the controls from the threat model and the integrity of the test system itself rather than relying on code review alone.

TDD and test-integrity requirements

Security defects begin with a regression test that demonstrably fails for the vulnerable behavior before the fix is implemented.
New security controls begin with failing invariant or abuse-case tests before production code.
Required tests cannot be skipped, disabled, marked expected-failure, weakened, or filtered out merely to obtain a green build.
Test harnesses, fuzz wrappers, subprocess launchers, and CI jobs must propagate assertion failures, crashes, signals, sanitizer findings, timeouts, corpus failures, and setup/teardown failures as non-zero.
Zero discovered or zero executed required tests is a failure.
Prohibit false-green constructs such as || true, || :, unconditional exit 0, swallowed exceptions, continue-on-error, allow_failure, and pass-with-no-tests behavior.
Add meta-tests that inject known failures into each security-test path and verify that the top-level command and CI job fail.

Test areas

Capture-off and stale-work cancellation invariants.
Redaction corpus, entropy detection, image-region masking, and metadata filtering.
Encryption envelope tampering, key failure, rotation, and GPG fallback.
Storage and log plaintext scanning.
IPC authentication and authorization.
Configuration, regex, parser, archive, and script-adapter fuzzing.
Remote-provider egress deny/allow behavior.
Symlink, permissions, race, path traversal, and crash-recovery tests.
Test-runner and wrapper failure propagation.

Acceptance criteria

Privacy invariants run in CI using synthetic data.
A seeded secret cannot appear in persisted files, logs, diagnostic bundles, exports, model requests, or test artifacts.
Fuzz targets have deterministic seeds and documented reproduction commands.
A seeded assertion failure, sanitizer finding, crash, timeout, and empty corpus/test selection each fail the canonical security-test command.
No required security test is skipped or marked expected-failure at release time.
Critical invariant or test-integrity failures block merges and releases.

Depends on

## Goal Continuously prove the controls from the threat model and the integrity of the test system itself rather than relying on code review alone. ## TDD and test-integrity requirements - Security defects begin with a regression test that demonstrably fails for the vulnerable behavior before the fix is implemented. - New security controls begin with failing invariant or abuse-case tests before production code. - Required tests cannot be skipped, disabled, marked expected-failure, weakened, or filtered out merely to obtain a green build. - Test harnesses, fuzz wrappers, subprocess launchers, and CI jobs must propagate assertion failures, crashes, signals, sanitizer findings, timeouts, corpus failures, and setup/teardown failures as non-zero. - Zero discovered or zero executed required tests is a failure. - Prohibit false-green constructs such as `|| true`, `|| :`, unconditional `exit 0`, swallowed exceptions, `continue-on-error`, `allow_failure`, and pass-with-no-tests behavior. - Add meta-tests that inject known failures into each security-test path and verify that the top-level command and CI job fail. ## Test areas - Capture-off and stale-work cancellation invariants. - Redaction corpus, entropy detection, image-region masking, and metadata filtering. - Encryption envelope tampering, key failure, rotation, and GPG fallback. - Storage and log plaintext scanning. - IPC authentication and authorization. - Configuration, regex, parser, archive, and script-adapter fuzzing. - Remote-provider egress deny/allow behavior. - Symlink, permissions, race, path traversal, and crash-recovery tests. - Test-runner and wrapper failure propagation. ## Acceptance criteria - Privacy invariants run in CI using synthetic data. - A seeded secret cannot appear in persisted files, logs, diagnostic bundles, exports, model requests, or test artifacts. - Fuzz targets have deterministic seeds and documented reproduction commands. - A seeded assertion failure, sanitizer finding, crash, timeout, and empty corpus/test selection each fail the canonical security-test command. - No required security test is skipped or marked expected-failure at release time. - Critical invariant or test-integrity failures block merges and releases. ## Depends on - #2 - #4 - #9 - #10 - #11 - #12 - #23 - #29 - #32 - #34

lost-rob0t commented

2026-08-30 09:24:25 +00:00

(Migrated from github.com)

Implemented and merged via PR #96 (squash commit on main). All acceptance criteria verified: CI privacy invariants on synthetic data, seeded-secret containment across storage/logs/archives/bundles (real crypto path byte-scans), deterministic fuzz targets with documented reproduction commands (docs/testing.md), nine failure-propagation modes including zero-selection, no skips/xfail (meta-tested check-policy patterns), and encode+decode IPC delete-capability enforcement. Full suite 1088 passing; canonical gate green. RAGE: rage/issue-38-security-fuzz-integrity.org.