Implement time-scoped retrieval and record provenance #77

Merged

lost-rob0t merged 39 commits from agent/issue-24-time-scoped-retrieval into main

2026-08-22 15:16:47 +00:00

lost-rob0t commented

2026-08-22 14:33:59 +00:00

(Migrated from github.com)

Closes #24.

RAGE scope

Implements time-scoped retrieval and record provenance from immutable starting main 6355587e905de3b2e6b2c71199d07eaaa4c557e2.

Implemented

deterministic timezone-aware resolution for explicit ISO dates, today/yesterday, bare weekdays, and bounded last N minutes|hours|days selectors;
DST-correct local-calendar half-open intervals using the configured IANA timezone;
coarse encrypted-storage day-bucket candidate selection without expanding the plaintext catalog;
optional semantic narrowing against IDs that still exist in canonical storage;
exact timestamp/application/workspace/metadata/keyword filtering over decrypted redacted records in memory;
query-time and per-record policy rechecks with provider eligibility carried only as control state;
bounded provenance-carrying passages with record IDs, capture timestamps, redaction revision/finding count, source IDs, observation timestamps, adapter revisions, and confidence;
deterministic semantic/keyword/provenance-confidence ranking with timestamp/record-ID tie breaking;
cancellation propagation with no partial-success batch;
focused retrieval documentation and README linkage.

Privacy/security boundary

The existing #11 plaintext catalog leakage budget is preserved. Exact timestamps, application/workspace metadata, OCR, and titles are not added to SQLite. Coarse UTC day buckets and encrypted semantic hits may narrow candidates; exact sensitive filtering happens only over bounded decrypted redacted records in memory. Retrieval cannot call providers or mint remote authorization, and returned passages expose no screenshot pixel field.

TDD / adversarial audit

Audit of earlier green head d1603293174ecb9d22da93d37076cc9a59f8a802 found a real acceptance gap: provenance confidence was carried but did not participate in plain-result ranking. Test-only head 7a76738287c617ea47931fb8585ac7c0438d40df established RED in CI run 514 (32580377293). Production now uses semantic score when present, keyword evidence for keyword queries, otherwise metadata provenance confidence.

Additional acceptance coverage includes a real SQLite encrypted-storage integration path, cancellation behavior, and security-layer content-sanitization fixtures. Synthetic secret markers use only per-line repository-supported pragma: allowlist secret; the secret baseline and scanner were not weakened.

Exact-head verification

Final head: f63afb9c674b1173a617b44f81d562722f58dbf4.

CI run 530 (32581103937) is fully green on that exact head:

Checks Ubuntu 22.04: success;
Checks Ubuntu 24.04: success;
Canonical ./scripts/check: success;
Failure propagation: success;
strict Pyright: 0 errors/warnings;
550 unit tests passed;
6 contract tests passed;
36 integration tests passed;
55 security tests passed;
formatting, Ruff, ShellCheck, repository policy, Bandit/detect-secrets/security scanning: passed.

No flake/package/config input changed, so this issue does not trigger the conditional nix flake check gate documented in the RAGE ledger.

Closes #24. ## RAGE scope Implements time-scoped retrieval and record provenance from immutable starting `main` `6355587e905de3b2e6b2c71199d07eaaa4c557e2`. ## Implemented - deterministic timezone-aware resolution for explicit ISO dates, today/yesterday, bare weekdays, and bounded `last N minutes|hours|days` selectors; - DST-correct local-calendar half-open intervals using the configured IANA timezone; - coarse encrypted-storage day-bucket candidate selection without expanding the plaintext catalog; - optional semantic narrowing against IDs that still exist in canonical storage; - exact timestamp/application/workspace/metadata/keyword filtering over decrypted redacted records in memory; - query-time and per-record policy rechecks with provider eligibility carried only as control state; - bounded provenance-carrying passages with record IDs, capture timestamps, redaction revision/finding count, source IDs, observation timestamps, adapter revisions, and confidence; - deterministic semantic/keyword/provenance-confidence ranking with timestamp/record-ID tie breaking; - cancellation propagation with no partial-success batch; - focused retrieval documentation and README linkage. ## Privacy/security boundary The existing #11 plaintext catalog leakage budget is preserved. Exact timestamps, application/workspace metadata, OCR, and titles are not added to SQLite. Coarse UTC day buckets and encrypted semantic hits may narrow candidates; exact sensitive filtering happens only over bounded decrypted redacted records in memory. Retrieval cannot call providers or mint remote authorization, and returned passages expose no screenshot pixel field. ## TDD / adversarial audit Audit of earlier green head `d1603293174ecb9d22da93d37076cc9a59f8a802` found a real acceptance gap: provenance confidence was carried but did not participate in plain-result ranking. Test-only head `7a76738287c617ea47931fb8585ac7c0438d40df` established RED in CI run 514 (`32580377293`). Production now uses semantic score when present, keyword evidence for keyword queries, otherwise metadata provenance confidence. Additional acceptance coverage includes a real SQLite encrypted-storage integration path, cancellation behavior, and security-layer content-sanitization fixtures. Synthetic secret markers use only per-line repository-supported `pragma: allowlist secret`; the secret baseline and scanner were not weakened. ## Exact-head verification Final head: `f63afb9c674b1173a617b44f81d562722f58dbf4`. CI run 530 (`32581103937`) is fully green on that exact head: - Checks Ubuntu 22.04: success; - Checks Ubuntu 24.04: success; - Canonical `./scripts/check`: success; - Failure propagation: success; - strict Pyright: 0 errors/warnings; - 550 unit tests passed; - 6 contract tests passed; - 36 integration tests passed; - 55 security tests passed; - formatting, Ruff, ShellCheck, repository policy, Bandit/detect-secrets/security scanning: passed. No flake/package/config input changed, so this issue does not trigger the conditional `nix flake check` gate documented in the RAGE ledger.