Implement authenticated encryption and key providers #54

Merged

lost-rob0t merged 77 commits from agent/issue-10-crypto-key-strategies into main

2026-07-18 20:10:04 +00:00

lost-rob0t commented

2026-07-18 18:00:15 +00:00

(Migrated from github.com)

What changed

Added XChaCha20-Poly1305-IETF authenticated encryption through PyNaCl/libsodium.
Added a versioned persistent envelope binding record ID, capture generation, configuration revision, creation time, algorithm, schema, and exact frame sizes as associated data.
Added one random data-encryption key per record and zeroed mutable plaintext/key buffers after use.
Added an OS-keyring strategy with versioned master keys, health reporting, wrapping/unwrapping, rotation, and revocation.
Added an explicitly configured GPG wrapping strategy using bounded async process execution, standard input/output only, finite timeouts, recipient health checks, and no temporary key files.
Added a provider registry that never selects a fallback unless its exact provider ID is configured and health checked.
Added ciphertext-preserving DEK rewrap so records can move to a new master-key version/provider without decrypting record content or writing plaintext files.
Added a strict encrypted-stage codec and a redacted-to-encrypted pipeline processor with cancellation checks and sanitized failures.
Extended the validated configuration schema for the implemented algorithm and explicit GPG fallback settings; effective inspection hides the recipient.
Updated key-provider protocols, domain envelopes, compatibility fixtures, configuration docs, and encryption/recovery documentation.

Security behavior

Encryption consumes RedactedStageItem directly in memory and produces EncryptedStageItem only.
Storage interfaces remain encrypted-envelope-only.
Ciphertext, nonce, authenticated metadata, wrong provider keys, malformed wrapped keys, and frame-size tampering fail authentication.
Missing, locked, invalid, unavailable, or revoked keys produce no encrypted output.
GPG is not inferred from availability and cannot be selected without fallback_provider_id = "gpg" plus a configured recipient.
GPG execution uses no shell, has a finite timeout, kills timed-out processes, ignores stderr content, and writes no plaintext temporary file.
Key IDs and versions may be persisted; DEKs, master keys, GPG plaintext output, subprocess stderr, and record plaintext are excluded from logs and configuration inspection.
Rotation retains old key versions until records are rewrapped; deliberate revocation makes remaining records on that version unrecoverable.

TDD and defect evidence

Tests cover:

authenticated round trips;
ciphertext and associated-metadata tampering;
wrong master keys;
keyring health, locked state, rotation, old-version recovery, and revocation;
GPG health, wrapping, associated-data mismatch, and explicit-only fallback;
ciphertext-preserving rewrap;
strict encrypted-stage framing;
cancellation before key creation;
missing-provider failure before encrypted output;
secret-free representations and effective configuration.

The development loop exposed and fixed:

keyring health probes incorrectly suppressing authorized first-key creation;
direct subprocess usage rejected by Bandit, replaced with bounded asyncio.create_subprocess_exec;
an overbroad sanitization assertion that rejected the intended fixed error code;
strict Pyright diagnostics around the PyNaCl binding surface;
GPG test and provider formatting under the repository's configured 100-column Ruff profile.

CPython 3.14 CI validation

Final workflow run #120 passed completely on head c188cb6f689cb3d357b4e0f1d0caedc1586ec374:

Ubuntu 22.04: dependency lock, Ruff format/lint, ShellCheck, strict Pyright, unit tests, contract tests, integration tests, security tests, and the combined repository policy/Bandit/detect-secrets scan passed.
Ubuntu 24.04: the same complete gate passed.
Independent eight-mode failure propagation passed.
The repository's standard CI workflow is unchanged from main.

Acceptance criteria

Storage APIs accept encrypted envelopes only.
Tampering and wrong keys fail authentication cleanly.
GPG fallback is explicit and health checked, never silently selected.
OS-keyring and explicit GPG strategies are available.
Rotation can rewrap DEKs without changing record ciphertext or creating plaintext files.
Key IDs and versions remain non-secret metadata; key material is never logged or configured.

Closes #10

## What changed - Added XChaCha20-Poly1305-IETF authenticated encryption through PyNaCl/libsodium. - Added a versioned persistent envelope binding record ID, capture generation, configuration revision, creation time, algorithm, schema, and exact frame sizes as associated data. - Added one random data-encryption key per record and zeroed mutable plaintext/key buffers after use. - Added an OS-keyring strategy with versioned master keys, health reporting, wrapping/unwrapping, rotation, and revocation. - Added an explicitly configured GPG wrapping strategy using bounded async process execution, standard input/output only, finite timeouts, recipient health checks, and no temporary key files. - Added a provider registry that never selects a fallback unless its exact provider ID is configured and health checked. - Added ciphertext-preserving DEK rewrap so records can move to a new master-key version/provider without decrypting record content or writing plaintext files. - Added a strict encrypted-stage codec and a redacted-to-encrypted pipeline processor with cancellation checks and sanitized failures. - Extended the validated configuration schema for the implemented algorithm and explicit GPG fallback settings; effective inspection hides the recipient. - Updated key-provider protocols, domain envelopes, compatibility fixtures, configuration docs, and encryption/recovery documentation. ## Security behavior - Encryption consumes `RedactedStageItem` directly in memory and produces `EncryptedStageItem` only. - Storage interfaces remain encrypted-envelope-only. - Ciphertext, nonce, authenticated metadata, wrong provider keys, malformed wrapped keys, and frame-size tampering fail authentication. - Missing, locked, invalid, unavailable, or revoked keys produce no encrypted output. - GPG is not inferred from availability and cannot be selected without `fallback_provider_id = "gpg"` plus a configured recipient. - GPG execution uses no shell, has a finite timeout, kills timed-out processes, ignores stderr content, and writes no plaintext temporary file. - Key IDs and versions may be persisted; DEKs, master keys, GPG plaintext output, subprocess stderr, and record plaintext are excluded from logs and configuration inspection. - Rotation retains old key versions until records are rewrapped; deliberate revocation makes remaining records on that version unrecoverable. ## TDD and defect evidence Tests cover: - authenticated round trips; - ciphertext and associated-metadata tampering; - wrong master keys; - keyring health, locked state, rotation, old-version recovery, and revocation; - GPG health, wrapping, associated-data mismatch, and explicit-only fallback; - ciphertext-preserving rewrap; - strict encrypted-stage framing; - cancellation before key creation; - missing-provider failure before encrypted output; - secret-free representations and effective configuration. The development loop exposed and fixed: - keyring health probes incorrectly suppressing authorized first-key creation; - direct `subprocess` usage rejected by Bandit, replaced with bounded `asyncio.create_subprocess_exec`; - an overbroad sanitization assertion that rejected the intended fixed error code; - strict Pyright diagnostics around the PyNaCl binding surface; - GPG test and provider formatting under the repository's configured 100-column Ruff profile. ## CPython 3.14 CI validation Final workflow run #120 passed completely on head `c188cb6f689cb3d357b4e0f1d0caedc1586ec374`: - Ubuntu 22.04: dependency lock, Ruff format/lint, ShellCheck, strict Pyright, unit tests, contract tests, integration tests, security tests, and the combined repository policy/Bandit/detect-secrets scan passed. - Ubuntu 24.04: the same complete gate passed. - Independent eight-mode failure propagation passed. - The repository's standard CI workflow is unchanged from `main`. ## Acceptance criteria - Storage APIs accept encrypted envelopes only. - Tampering and wrong keys fail authentication cleanly. - GPG fallback is explicit and health checked, never silently selected. - OS-keyring and explicit GPG strategies are available. - Rotation can rewrap DEKs without changing record ciphertext or creating plaintext files. - Key IDs and versions remain non-secret metadata; key material is never logged or configured. Closes #10