Implement encrypted storage catalog and migrations #56

Closed

lost-rob0t wants to merge 45 commits from agent/issue-11-storage-catalog into main

lost-rob0t commented

2026-07-18 21:16:07 +00:00

(Migrated from github.com)

Iteration scope

Implements issue #11 as one clean vertical slice.

What changed

Added a complete outer authenticated-encryption layer around each existing record envelope so exact timestamps, configuration revisions, inner ciphertext, nonces, wrapped record keys, and frame metadata remain hidden from filesystem inspection.
Added a minimized owner-only SQLite routing catalog plus opaque encrypted .lre blob files.
Added exact arbitrary time-range retrieval. Coarse UTC day buckets select candidates only; exact encrypted timestamps are then used to answer intervals such as five minutes ago, eighteen hours ago, or sub-minute ranges.
Added recoverable pending transactions for atomic blob publication and schema replacement.
Added startup recovery for interrupted writes and orphan encrypted temporary files.
Added storage schema versioning and forward rewrite of supported prior blobs without plaintext migration files.
Added encrypted-byte and record-count quotas.
Added encrypted-blob digest checks and corruption quarantine.
Added runtime rejection of non-EncryptedRecordEnvelope payloads and unsafe symlink storage roots.
Preserved key-provider availability failures as recoverable key errors; a locked keyring does not misclassify or quarantine a valid encrypted blob.
Documented the storage layout, accepted metadata leakage, exact-time semantics, recovery, migration, quota, quarantine, and deletion limits.

Security behavior

SQLite never stores exact timestamps, titles, URLs, OCR, screenshots, summaries, prompts, embeddings, or model output.
A five-minute query remains exact: the day bucket is merely a candidate locator, while filtering uses decrypted timestamp precision down to Python datetime microseconds.
Temporary, pending, committed, migrated, orphaned, and quarantined artifacts contain encrypted bytes only.
Interrupted publication cannot produce a visible partial record; only committed catalog rows are queryable.
Unsupported future schemas, insecure paths, quota failures, digest mismatches, and authentication failures fail closed.
Locked or unavailable keys block reads without destroying or quarantining valid records.
Deletion does not falsely claim physical secure erase or cryptographic destruction.

Tests

Tests cover:

encrypted blob round trips and plaintext inspection;
authenticated tamper rejection;
prior-schema migration detection;
exact five-minute and eighteen-hour retrieval windows;
interrupted publication and startup completion;
quota rejection;
corruption quarantine;
locked-key recovery without false corruption;
encrypted-only runtime type enforcement;
symlink-root rejection;
deletion semantics.

Validation

Local validation against the repository's pinned tool versions and strict settings:

Ruff formatting: passed.
Ruff lint: passed.
Strict Pyright: 0 errors.
New unit, integration, and security tests: 12 passed.
Bandit on the new storage package: passed.
detect-secrets with the repository baseline on all changed Python files: passed.

GitHub Actions previously reached formatting, Ruff, ShellCheck, and Pyright during development. The final clean-head runs currently fail before runner setup with zero steps and no job logs, including the independent failure-propagation job. The PR remains draft until the required hosted CI can execute normally.

Closes #11

## Iteration scope Implements issue #11 as one clean vertical slice. ## What changed - Added a complete outer authenticated-encryption layer around each existing record envelope so exact timestamps, configuration revisions, inner ciphertext, nonces, wrapped record keys, and frame metadata remain hidden from filesystem inspection. - Added a minimized owner-only SQLite routing catalog plus opaque encrypted `.lre` blob files. - Added exact arbitrary time-range retrieval. Coarse UTC day buckets select candidates only; exact encrypted timestamps are then used to answer intervals such as five minutes ago, eighteen hours ago, or sub-minute ranges. - Added recoverable pending transactions for atomic blob publication and schema replacement. - Added startup recovery for interrupted writes and orphan encrypted temporary files. - Added storage schema versioning and forward rewrite of supported prior blobs without plaintext migration files. - Added encrypted-byte and record-count quotas. - Added encrypted-blob digest checks and corruption quarantine. - Added runtime rejection of non-`EncryptedRecordEnvelope` payloads and unsafe symlink storage roots. - Preserved key-provider availability failures as recoverable key errors; a locked keyring does not misclassify or quarantine a valid encrypted blob. - Documented the storage layout, accepted metadata leakage, exact-time semantics, recovery, migration, quota, quarantine, and deletion limits. ## Security behavior - SQLite never stores exact timestamps, titles, URLs, OCR, screenshots, summaries, prompts, embeddings, or model output. - A five-minute query remains exact: the day bucket is merely a candidate locator, while filtering uses decrypted timestamp precision down to Python `datetime` microseconds. - Temporary, pending, committed, migrated, orphaned, and quarantined artifacts contain encrypted bytes only. - Interrupted publication cannot produce a visible partial record; only `committed` catalog rows are queryable. - Unsupported future schemas, insecure paths, quota failures, digest mismatches, and authentication failures fail closed. - Locked or unavailable keys block reads without destroying or quarantining valid records. - Deletion does not falsely claim physical secure erase or cryptographic destruction. ## Tests Tests cover: - encrypted blob round trips and plaintext inspection; - authenticated tamper rejection; - prior-schema migration detection; - exact five-minute and eighteen-hour retrieval windows; - interrupted publication and startup completion; - quota rejection; - corruption quarantine; - locked-key recovery without false corruption; - encrypted-only runtime type enforcement; - symlink-root rejection; - deletion semantics. ## Validation Local validation against the repository's pinned tool versions and strict settings: - Ruff formatting: passed. - Ruff lint: passed. - Strict Pyright: 0 errors. - New unit, integration, and security tests: 12 passed. - Bandit on the new storage package: passed. - detect-secrets with the repository baseline on all changed Python files: passed. GitHub Actions previously reached formatting, Ruff, ShellCheck, and Pyright during development. The final clean-head runs currently fail before runner setup with zero steps and no job logs, including the independent failure-propagation job. The PR remains draft until the required hosted CI can execute normally. Closes #11

lost-rob0t commented

2026-07-18 22:37:40 +00:00

(Migrated from github.com)

Superseded by clean replacement PR #58. The replacement branch starts from merged main, contains no temporary workflow history, preserves provider-availability errors without quarantining valid blobs, and includes the corrected single-quarantine path.

Superseded by clean replacement PR #58. The replacement branch starts from merged `main`, contains no temporary workflow history, preserves provider-availability errors without quarantining valid blobs, and includes the corrected single-quarantine path.