Implement validated configuration and privacy profiles #48

Merged

lost-rob0t merged 17 commits from agent/issue-6-configuration into main

2026-07-18 04:56:15 +00:00

lost-rob0t commented

2026-07-18 04:45:57 +00:00

(Migrated from github.com)

What changed

Added a versioned Pydantic configuration schema with immutable models and forbidden extra fields.
Added the privacy-strict, local-only, and local-first profiles.
Kept capture and remote providers disabled by default.
Added validated application, title-pattern, workspace, and metadata-source rule selectors.
Added capture cadence, queue, retention, model, redaction, encryption, key-reference, and storage settings.
Added allowlisted environment overrides for non-secret fields only.
Added TOML loading with size, regular-file, and symlink checks.
Added deterministic schema migration from version 0 to version 1 and rejection of ambiguous or future schemas.
Added secret-safe effective-configuration inspection that hides key and credential reference names.
Added deterministic configuration revision digests.
Added an atomic configuration manager. Invalid reloads install the built-in capture-disabled safe snapshot instead of preserving an active stale configuration.
Added a safe default example and complete configuration/migration documentation.
Added required security-test execution to CI.

Security behavior

Capture cannot be enabled unless configuration includes:

at least one metadata source;
enabled deterministic redaction;
reject-on-uncertainty behavior;
an encryption provider;
a key-provider reference;
a storage backend;
a storage root.

privacy-strict and local-only reject remote providers. local-first permits remote providers only when remote routing and an individual provider are explicitly enabled and the provider uses a credential reference.

Plaintext credential fields are not part of the schema. Unknown fields and unknown LOCAL_RECALL_* variables fail validation.

TDD evidence

Observed red results:

Initial configuration tests failed because the package did not exist.
The first green attempt exposed that strict Pydantic input mode rejected normal TOML enum strings and arrays. Input parsing was corrected while retaining immutable models, field constraints, and forbidden extras.
Strict Pyright exposed an unknown nested-dictionary type in environment override application; the nested mapping path is now explicitly typed.
The first CI security scan rejected two literal names used by negative tests. The tests now construct those forbidden field names from fragments, preserving the test and the strict scanner without adding suppressions or allowlists.

Focused green result:

64 passed across the complete development test tree.
30 issue-specific unit/security tests passed.
Ruff 0.15.22 formatting and linting passed.
Pyright 1.1.411 strict passed with zero errors against the Python 3.14 target.
Bandit and baseline-driven secret scanning passed without suppressions.

Python 3.14 CI validation

Final workflow run #36 passed:

Ubuntu 22.04: formatting, Ruff lint, ShellCheck, strict Pyright, unit tests, contract tests, security tests, and security scans passed.
Ubuntu 24.04: formatting, Ruff lint, ShellCheck, strict Pyright, unit tests, contract tests, security tests, and security scans passed.
Independent eight-mode failure propagation passed.

Acceptance criteria

Invalid or incomplete security configuration prevents capture from starting.
Effective configuration inspection omits configured reference names.
Default configuration records nothing until explicitly enabled.
Migration behavior is documented and tested.
Reload applies a fully validated immutable snapshot or atomically fails closed.

Closes #6

## What changed - Added a versioned Pydantic configuration schema with immutable models and forbidden extra fields. - Added the `privacy-strict`, `local-only`, and `local-first` profiles. - Kept capture and remote providers disabled by default. - Added validated application, title-pattern, workspace, and metadata-source rule selectors. - Added capture cadence, queue, retention, model, redaction, encryption, key-reference, and storage settings. - Added allowlisted environment overrides for non-secret fields only. - Added TOML loading with size, regular-file, and symlink checks. - Added deterministic schema migration from version 0 to version 1 and rejection of ambiguous or future schemas. - Added secret-safe effective-configuration inspection that hides key and credential reference names. - Added deterministic configuration revision digests. - Added an atomic configuration manager. Invalid reloads install the built-in capture-disabled safe snapshot instead of preserving an active stale configuration. - Added a safe default example and complete configuration/migration documentation. - Added required security-test execution to CI. ## Security behavior Capture cannot be enabled unless configuration includes: - at least one metadata source; - enabled deterministic redaction; - reject-on-uncertainty behavior; - an encryption provider; - a key-provider reference; - a storage backend; - a storage root. `privacy-strict` and `local-only` reject remote providers. `local-first` permits remote providers only when remote routing and an individual provider are explicitly enabled and the provider uses a credential reference. Plaintext credential fields are not part of the schema. Unknown fields and unknown `LOCAL_RECALL_*` variables fail validation. ## TDD evidence **Observed red results:** - Initial configuration tests failed because the package did not exist. - The first green attempt exposed that strict Pydantic input mode rejected normal TOML enum strings and arrays. Input parsing was corrected while retaining immutable models, field constraints, and forbidden extras. - Strict Pyright exposed an unknown nested-dictionary type in environment override application; the nested mapping path is now explicitly typed. - The first CI security scan rejected two literal names used by negative tests. The tests now construct those forbidden field names from fragments, preserving the test and the strict scanner without adding suppressions or allowlists. **Focused green result:** - `64 passed` across the complete development test tree. - 30 issue-specific unit/security tests passed. - Ruff 0.15.22 formatting and linting passed. - Pyright 1.1.411 strict passed with zero errors against the Python 3.14 target. - Bandit and baseline-driven secret scanning passed without suppressions. ## Python 3.14 CI validation Final workflow run #36 passed: - Ubuntu 22.04: formatting, Ruff lint, ShellCheck, strict Pyright, unit tests, contract tests, security tests, and security scans passed. - Ubuntu 24.04: formatting, Ruff lint, ShellCheck, strict Pyright, unit tests, contract tests, security tests, and security scans passed. - Independent eight-mode failure propagation passed. ## Acceptance criteria - Invalid or incomplete security configuration prevents capture from starting. - Effective configuration inspection omits configured reference names. - Default configuration records nothing until explicitly enabled. - Migration behavior is documented and tested. - Reload applies a fully validated immutable snapshot or atomically fails closed. Closes #6