[P2] Add end-to-end acceptance, reliability, and performance tests #39

New issue

Closed

opened 2026-07-18 00:04:23 +00:00 by lost-rob0t · 1 comment

lost-rob0t commented

2026-07-18 00:04:23 +00:00

(Migrated from github.com)

Goal

Validate the complete system under realistic desktop and failure scenarios while proving that end-to-end failures cannot be reported as success.

TDD and execution requirements

Each E2E scenario is introduced as a failing acceptance test before the corresponding feature path is considered complete.
Bug fixes add a failing E2E regression when the defect crosses component boundaries.
Scenario runners must propagate failures from every child process, pipeline stage, container/VM command, timeout, assertion, and teardown check.
Zero selected, discovered, or executed required scenarios is a failure.
No scenario may use || true, unconditional success exits, swallowed exceptions, continue-on-error, retry-until-green without a final failing status, or equivalent escape routes.
Retries are allowed only for explicitly classified transient operations; exhaustion must fail non-zero and report the original cause.
Add self-tests that intentionally fail one scenario at a time and verify the top-level E2E command and CI job fail.

Scenarios

Start, record, pause, privacy mode, stop, restart, and query.
Qtile metadata plus generic Xorg fallback.
Lock/unlock, idle/resume, monitor changes, model outage, key lock, low disk, and corrupted index.
Capture synthetic work sessions and answer “What was I doing Saturday?” with correct citations.
Long-running soak test with bounded queues and retention enabled.
Offline local-only operation.

Performance targets

Define acceptable capture latency, idle CPU, peak memory, storage growth, OCR throughput, query latency, and model resource budgets for an 8B-class setup.
Performance-threshold violations fail the relevant required job; they are not warnings disguised as success.

Acceptance criteria

E2E tests use isolated synthetic desktops and never inspect the developer’s real screen.
Resource usage remains bounded during soak and failure tests.
Stop/privacy mode halts new persistence within a defined measurable bound.
A seeded assertion failure, process crash, timeout, teardown failure, and empty scenario selection each fail the canonical E2E command.
Required scenarios contain no skipped or expected-failure state at release time.
A release report records passed scenarios, exact failures, and known limitations without relabeling failures as passes.

Depends on

## Goal Validate the complete system under realistic desktop and failure scenarios while proving that end-to-end failures cannot be reported as success. ## TDD and execution requirements - Each E2E scenario is introduced as a failing acceptance test before the corresponding feature path is considered complete. - Bug fixes add a failing E2E regression when the defect crosses component boundaries. - Scenario runners must propagate failures from every child process, pipeline stage, container/VM command, timeout, assertion, and teardown check. - Zero selected, discovered, or executed required scenarios is a failure. - No scenario may use `|| true`, unconditional success exits, swallowed exceptions, `continue-on-error`, retry-until-green without a final failing status, or equivalent escape routes. - Retries are allowed only for explicitly classified transient operations; exhaustion must fail non-zero and report the original cause. - Add self-tests that intentionally fail one scenario at a time and verify the top-level E2E command and CI job fail. ## Scenarios - Start, record, pause, privacy mode, stop, restart, and query. - Qtile metadata plus generic Xorg fallback. - Lock/unlock, idle/resume, monitor changes, model outage, key lock, low disk, and corrupted index. - Capture synthetic work sessions and answer “What was I doing Saturday?” with correct citations. - Long-running soak test with bounded queues and retention enabled. - Offline local-only operation. ## Performance targets - Define acceptable capture latency, idle CPU, peak memory, storage growth, OCR throughput, query latency, and model resource budgets for an 8B-class setup. - Performance-threshold violations fail the relevant required job; they are not warnings disguised as success. ## Acceptance criteria - E2E tests use isolated synthetic desktops and never inspect the developer’s real screen. - Resource usage remains bounded during soak and failure tests. - Stop/privacy mode halts new persistence within a defined measurable bound. - A seeded assertion failure, process crash, timeout, teardown failure, and empty scenario selection each fail the canonical E2E command. - Required scenarios contain no skipped or expected-failure state at release time. - A release report records passed scenarios, exact failures, and known limitations without relabeling failures as passes. ## Depends on - #4 - #26 - #28 - #31 - #37 - #38

lost-rob0t commented

2026-08-30 10:22:37 +00:00

(Migrated from github.com)

Implemented and merged via PR #97. All acceptance criteria verified: 12 E2E scenarios over isolated synthetic desktops through the real crypto/storage/index/retrieval/answering/health stack; bounded soak with retention; stop/persistence halt bounds (<2s); capture/query latency and storage-growth budgets enforced as failing assertions; six failure-injection self-tests (scripts/verify-e2e-failure-modes) wired into the canonical gate and the CI failure-propagation job; zero-selection fails; no skips/xfail; offline local-only scenario included. Full suite 1100 passing; canonical gate green. Performance budgets for CI documented in docs/e2e.md; reference-hardware (8B-class) budgets recorded as release-gate items. RAGE: rage/issue-39-e2e-reliability-performance.org.