# Regression Test Suite

> **Status:** Implemented. All tiers are live — infrastructure, functional regression, real-OS attachment/SAF tests, lifecycle/robustness tests, billing sandbox, and macrobenchmarks.
>
> **Companion docs:** [plan-smoke-tests.md](plan-smoke-tests.md) (PR-gated smoke suite), [specs.md](specs.md) (ACs the regression suite covers), [TESTING_SETUP.md](../TESTING_SETUP.md) (GMD / Robolectric infrastructure), [billing-sandbox.md](billing-sandbox.md) (Play Console / CI runbook), [regression-dashboard.md](regression-dashboard.md) (nightly health), [regression-promotion-playbook.md](regression-promotion-playbook.md) (tier promotion/demotion rules).

## 1. Goals & principles

- **Catch real-device bugs before release** — SAF, camera, FileProvider, billing, lifecycle, perf. Things JVM/Robolectric structurally can't catch.
- **Functional regression, not duplication** — don't re-test what `*Test.kt` / `*IT.kt` / `*ScreenshotTest.kt` already cover deterministically on JVM. The regression suite owns *behavior that depends on the real OS*.
- **Independent tests, deterministic setup** — every scenario resets DataStore + vault state (via `resetYorvanaState` / `installFreshVault`). No accumulated state across tests.
- **Annotation-tiered, single source tree** — tests live in `app/src/androidTest`, custom marker annotations control which tests run in PR-gated vs nightly via AndroidJUnitRunner `annotation` filters.
- **Flake budget: zero on PR-gated tier, ≤1% on nightly** — quarantine, don't ignore. Flaky tests get a `@Quarantined` marker annotation; PR-gated and core regression tasks pass `notAnnotation=com.yorvana.testsupport.tiers.Quarantined` so these tests are excluded from gating, while the nightly `regressionFullCheck` task does *not* set `notAnnotation` so they still execute and report.

## 2. Technical approach

Built on the existing stack — `AndroidJUnit4` + `createAndroidComposeRule<MainActivity>()` + GMD Pixel 2 API 33 — with these additions:

| Concern | Tool | Where it lives |
|---|---|---|
| Real SAF picker, real file picker, real camera | **UiAutomator** (`UiDevice`) | `androidTest/.../regression/setup/`, `regression/vault/`, `regression/attachments/` |
| Process death | `ProcessDeath.recreate()` / `ProcessDeath.kill()` (via `am kill`, never `force-stop`) | `regression/lifecycle/` |
| Configuration changes | `composeTestRule.activityRule.scenario.recreate()` | `regression/lifecycle/` |
| Google Play Billing sandbox | External `billing-sandbox.yml` workflow on a self-hosted runner with a real Play-enabled device; multi-precondition `Assume.assumeTrue` skip guard | `regression/billing/` |
| Performance NFRs | **Macrobenchmark** (`androidx.benchmark`) in the `:macrobenchmark` module, with a dedicated GMD entry generating `:macrobenchmark:pixel2api33BenchmarkAndroidTest` | `:macrobenchmark` module |
| Fixture data | Minimal bundled assets (`app/src/androidTest/assets/`); large datasets generated in-process by test helpers | `testsupport/` helpers |

### Test tiers (custom marker annotations)

Six runtime-retained marker annotations live in `app/src/androidTest/.../testsupport/tiers/`, all carrying `@java.lang.annotation.Inherited`:

| Annotation | Purpose | Targets |
|---|---|---|
| `@Smoke` | Existing PR-gated smoke tests | `CLASS`, `FUNCTION` |
| `@Regression` | Core regression — PR-gated (label-gated) + nightly | `CLASS`, `FUNCTION` |
| `@RegressionFull` | Extended regression — nightly only | `CLASS`, `FUNCTION` |
| `@BillingSandbox` | Billing sandbox — nightly on self-hosted runner | `CLASS`, `FUNCTION` |
| `@Quarantined` | Flaky test marker with `since`, `issue`, `reason` params | `CLASS`, `FUNCTION` |
| `@ScenarioId` | Stable spec ID (e.g. `@ScenarioId("R-A01")`) for dashboard tracking | `FUNCTION` |

### Tier budget

| Tier | Marker | Runs | Budget |
|---|---|---|---|
| Smoke | `@Smoke` | PR (label-gated) | <2 min |
| Regression — core | `@Regression` | PR (label-gated) + nightly | ~15 min |
| Regression — full | `@RegressionFull` | Nightly only | ~45–60 min |
| Billing sandbox | `@BillingSandbox` | Nightly on self-hosted runner | ~10 min |
| Macrobenchmark | Separate `:macrobenchmark` module | Nightly | ~15 min |

### Gradle tasks

Regression tasks use **`GradleBuild` child invocations** — each registers a `GradleBuild` task via `registerRegressionDeviceTask()` that runs `:app:pixel2api33DebugAndroidTest` in a child Gradle build with its own runner-argument set. This ensures `-P` annotation values do not collide across tiers.

- **`regressionCheck`** — PR/core tier. `annotation=Regression`, `notAnnotation=Quarantined`. Excludes `RegressionFull` and `BillingSandbox` (neither carries `@Regression`).
- **`regressionFullCheck`** — Nightly aggregator. Depends on `regressionFullCoreCheck`, `regressionFullExtendedCheck`, and `:macrobenchmark:pixel2api33BenchmarkAndroidTest`. Does *not* `dependsOn(regressionCheck)` because that task's `notAnnotation=Quarantined` filter would hide quarantined core tests from nightly (contradicting the quarantine policy).
  - `regressionFullCoreCheck` — `annotation=Regression`, **no** `notAnnotation`. Flaky `@Regression @Quarantined` tests still execute and report.
  - `regressionFullExtendedCheck` — `annotation=RegressionFull`.
  - `preserveRegressionCoreResults` — `Copy` task that saves `@Regression` XML results before `regressionFullExtendedCheck` overwrites the shared `managedDevice/` output directory. Wired via `finalizedBy`/`mustRunAfter`.
- **Billing sandbox** — runs via a separate `billing-sandbox.yml` workflow on a self-hosted runner, outside `regressionFullCheck`.
- **Smoke isolation** — the `pixel2api33` device's `instrumentationRunnerArguments` applies `annotation=Smoke` (alongside `notPackage=com.yorvana.screenshots`) when not in an explicit annotation run. This positive filter prevents regression tests from leaking into the smoke budget. Injection is scoped by exact device/task-name match so future Play/staging device tasks do not inherit the smoke filter.

### Annotation-misuse guardrail

A source-level scanner (`RegressionTierScanner`) in `app/src/test/.../testsupport/RegressionTierScannerTest.kt` validates that:
- Every `@Test` method under `com.yorvana.regression` has exactly one tier annotation (at method or class level, including `@Inherited` superclass resolution).
- `@Quarantined` is never orphaned without a tier annotation.
- `@Quarantined` parameters follow the required format (`since` as ISO-8601 date, `issue` as issue number, `reason` as description).

Negative-test fixtures cover: zero tiers, two tiers, orphan quarantine, inherited-tier sanity, invalid quarantine format.

### CI workflows

- **`regression.yml`** — `workflow_call` + `workflow_dispatch` + `schedule` (daily cron). Runs `regressionCheck` on label/manual/reusable paths; `regressionFullCheck` on the nightly cron path. Mirrors `smoke.yml`'s KVM setup, forces `swiftshader_indirect`. Uploads managed-device reports, regression-core-preserved results, and macrobenchmark reports. The `aggregate-results` job runs `tools/aggregate_regression.py` and auto-commits updated `regression-dashboard.md` + `regression-history.json`.
- **`regression-label.yml`** — `pull_request: [labeled]`, gated on the `regression` label. Calls `regression.yml` as a reusable workflow.
- **`billing-sandbox.yml`** — `workflow_dispatch` (with `scenario` choice) + `schedule` (nightly). Runs on `[self-hosted, android-billing-sandbox]` with a real Play-enabled device. Builds a `staging` variant signed with the upload key, runs `connectedStagingAndroidTest` with `annotation=BillingSandbox`.

## 3. Test coverage map

Mapped to [specs.md](specs.md) ACs. The principle: every AC has either a JVM test (existing) **or** a regression scenario, with regression earning its place by needing real OS behavior.

### 3.1 Vehicle CRUD (FR-V1..V4)

All tests in `regression/vehicles/VehicleRegressionTest.kt`, class-level `@Regression`.

| ID | Scenario | Key assertions |
|---|---|---|
| **R-V01** | Add vehicle with all optional fields, kill process, relaunch | Vehicle present with all fields on disk (vault JSON readable) |
| **R-V02** | Duplicate VIN guard | Second vehicle with same VIN blocked with error |
| **R-V03** | Field length / range validation (nickname >50, year=1899, year=2101) | Save blocked, error visible |
| **R-V04** | Edit vehicle, rotate device mid-edit | Form state preserved (`rememberSaveable`) |
| **R-V05** | Delete vehicle with 50 records + 20 attachments | All files removed from file-backed vault tree |
| **R-V06** | External vault edit: pre-seed vehicle file directly, relaunch | Vehicle appears (catches cache-staleness bugs) |
| **R-V07** | VIN copy-to-clipboard | `ClipboardManager.primaryClip` matches the VIN |

### 3.2 Records (FR-R1..R4)

All tests in `regression/records/RecordRegressionTest.kt`, class-level `@Regression`.

| ID | Scenario | Key assertions |
|---|---|---|
| **R-R01** | Add record, kill app mid-form | No partial record on disk (transactional save) |
| **R-R02** | Odometer/cost boundary validation (0, max, max+1, decimals) | Appropriate error messages for each boundary |
| **R-R03** | 200-record list correctness | Scroll reaches first and last records, order stable, no crash |
| **R-R04** | Delete record | Confirm attachment files gone from disk |

### 3.3 Attachments (FR-A1, FR-A2)

Tests split across two files:

- `regression/attachments/AttachmentRealOsRegressionTest.kt` — class-level `@RegressionFull`
- `regression/attachments/ImageViewerGestureRegressionTest.kt` — class-level `@Regression`

The app uses `ActivityResultContracts.TakePicture()` with a `FileProvider` output URI and does **not** declare or request the `CAMERA` runtime permission. Camera scenarios do not assert a permission dialog. External-viewer assertions use a resolver-presence gate: `ACTION_VIEW` only asserts external UI when `packageManager.resolveActivity(…) != null`.

| ID | Tier | Scenario | Key assertions |
|---|---|---|---|
| **R-A01** | `RegressionFull` | Real SAF file picker via UiAutomator, pick PDF | Attachment persists across kill, URI resolves, `ACTION_VIEW` fires |
| **R-A02** | `RegressionFull` | Real camera capture via `TakePicture()` + `FileProvider` | Photo persists as JPG in vault |
| **R-A03** | `Regression` | Image viewer gestures (pinch zoom, double-tap toggle, pan, zoom clamp) | User-observable rendering assertions via `boundsInRoot`; no reads of private scale/offset state |
| **R-A04** | `RegressionFull` | Non-image attachment tap | System viewer intent launches (resolver-presence gate) |
| **R-A05** | `RegressionFull` | Remove attachment with confirmation | File deleted from vault fixture |
| **R-A06** | `RegressionFull` | Vault folder change mid-session | Attachment still resolvable after move |
| **R-A07** | `RegressionFull` | Cancelled camera capture recovery | Clear error, no partial JPG, retry succeeds |

### 3.4 Vault / SAF (FR-D1)

Tests split across two files:

- `regression/setup/SetupRegressionTest.kt` — class-level `@Regression` (R-D01)
- `regression/vault/VaultRealOsRegressionTest.kt` — class-level `@RegressionFull` (R-D02..D04)

| ID | Tier | Scenario | Key assertions |
|---|---|---|---|
| **R-D01** | `Regression` | Fresh setup: drive real SAF folder picker via UiAutomator | Setup completes, garage screen loads |
| **R-D02** | `RegressionFull` | Pre-existing vault: point at folder with sample data | Vehicles + records load |
| **R-D03** | `RegressionFull` | Vault directory deleted externally between sessions | App does not crash on relaunch. *Note:* the plan originally called for testing SAF persistable URI permission revocation, but this was descoped due to SAF fragility on GMD emulators. The current test covers the "vault disappears" failure mode via directory deletion on a file-backed vault. |
| **R-D04** | `RegressionFull` | Malformed `vehicle.json` in vault | App skips that vehicle and does not crash the list |
| **R-D05** | — | Migration: pre-seed an old-format vault, verify upgrade | **Not yet implemented.** No migration logic exists because the app has not shipped to production yet — there is no legacy schema to migrate from. This scenario will be implemented when the first schema migration is needed. |

### 3.5 Categories (FR-C1, FR-C2)

Tests in `regression/settings/SettingsCategoriesPaywallRegressionTest.kt`, class-level `@Regression`.

| ID | Scenario | Key assertions |
|---|---|---|
| **R-C01** | Default categories present on fresh install | All 8 default categories displayed |
| **R-C02** | Add custom category, kill app, relaunch | Custom category persists, appears in selector across vehicles |
| **R-C03** | Delete custom category in use by records | Confirm prompt mentions affected record count |

### 3.6 Settings (FR-S1..S3)

- R-S01, R-S03, R-S04 in `regression/settings/SettingsCategoriesPaywallRegressionTest.kt` — class-level `@Regression`
- R-S02 (a/b/c) in `regression/vault/VaultRealOsRegressionTest.kt` — class-level `@RegressionFull` (grouped with vault tests since all three paths exercise the real SAF picker)

| ID | Tier | Scenario | Key assertions |
|---|---|---|---|
| **R-S01** | `Regression` | Toggle odometer unit | Existing records re-render with new unit |
| **R-S02a** | `RegressionFull` | Change vault folder — **Move** path | Byte-identical copy at destination, source deleted, reopen boots into new tree |
| **R-S02b** | `RegressionFull` | Change vault folder — **Start fresh** path | Source left untouched, new tree starts empty |
| **R-S02c** | `RegressionFull` | Change vault folder — **Abort/failure** path | Source intact, vault pointer unchanged, clear error (uses `VaultStorageTestHooks.failMoveAfterCopiedFiles` to inject failure) |
| **R-S03** | `Regression` | Crash reporting opt-in toggle | Persists across kill + restart |
| **R-S04** | `Regression` | Custom currency persistence | Custom currency code rendered on existing records and used for new records |

### 3.7 Paywall (FR-P1..P6)

| ID | Tier | Scenario | Notes |
|---|---|---|---|
| **R-P01** | — | Free tier, attempt 2nd vehicle → upgrade dialog | **Already covered** by `SmokeTest.upgradeDialogOnFreeTier`. No separate regression test. |
| **R-P02** | — | Read-only mode on free tier across screens | **Already covered** by `SmokeTest.readOnlyBannerAcrossScreens`. No separate regression test. |
| **R-P03** | `BillingSandbox` | Complete real test purchase via Play Billing | Full ADR-012 contract: paywall sheet → Play Billing → license-tester purchase → `isPremium=true`, sheet dismisses, app auto-navigates to Add Vehicle. Runs via `billing-sandbox.yml` on self-hosted runner. |
| **R-P04** | `BillingSandbox` | Restore purchases on fresh install (`pm clear`) | Asserts `false → restore → true → Settings re-renders` transition. |
| **R-P05** | `Regression` | Offline cached premium (not sandbox-dependent) | Seeds cache via `AppPreferencesStore.setIsPremiumCached(true)`. Goes offline via device-level `svc wifi disable` / `svc data disable`. Asserts `isPremium` resolves through `cachedPremium` (not `billingDerivedPremium`). `@After` restores connectivity unconditionally. |

### 3.8 Lifecycle & robustness

Tests in `regression/lifecycle/LifecycleRobustnessRegressionTest.kt` — method-level annotations (mixed tiers).

| ID | Tier | Scenario | Key assertions |
|---|---|---|---|
| **R-L01** | `Regression` | Recreate vehicle form (canonical core) | ViewModel-backed draft fields preserved; transient `remember`-only UI out of scope |
| **R-L01** | `RegressionFull` | Recreate record form (extended) | Same contract for record form draft state |
| **R-L02** | `Regression` | Hard kill on record detail | Route restoration + persisted data visible. Uses `ProcessDeath.kill()` (`am kill`, never `force-stop`). |
| **R-L03** | `Regression` | Low-memory simulation (`am send-trim-memory`) | No crash |
| **R-L04** | `Regression` | Back-press exhaustively from deep screens | Navigates through stack correctly |
| **R-L05** | `Regression` | Launcher re-entry while on deep screen | Returns to existing task/screen (task-affinity / `singleTask` behavior) |
| **R-L06** | `RegressionFull` | Disk-full / write failure on save | See below |
| **R-L07** | `Regression` | Concurrent external edit (backgrounded) | Resume reflects externally mutated `vehicle.json` |

**R-L06** detail — tested in `regression/vault/StorageRobustnessRegressionTest.kt`, class-level `@RegressionFull`:

Two file-backed tests:
1. Injected write failure (via `VaultStorageTestHooks.failNextJsonWriteAfterTempWrite`) → existing JSON is byte-identical to pre-write, no orphan `.tmp`, retry succeeds.
2. Orphan `.tmp` recovery: when active file is missing, a valid `.tmp` is promoted; when active file exists, stale `.tmp` is cleaned up.

*Note:* only the file-backed (`VaultNode.Local`) path is tested. The SAF (`VaultNode.Remote`) path — which has a weaker contract (no guaranteed atomic rename, narrower assertion: clear error + no orphan `.tmp` past next open) — is deferred as future coverage.

### 3.9 NFRs — Macrobenchmark module

Tests in `:macrobenchmark` module (`macrobenchmark/src/main/java/.../YorvanaBenchmark.kt`).

| ID | Metric | Scenario |
|---|---|---|
| **M-01** | `StartupTimingMetric` | Cold start with 20-vehicle fixture. P50 ≤ 2000ms is advisory. |
| **M-02** | `FrameTimingMetric` | Vehicle list scroll FPS with 20 vehicles |
| **M-03** | `FrameTimingMetric` | Record list scroll FPS with 200 records |
| **M-04** | `FrameTimingMetric` | Navigation journey garage→detail→record (scripted) |

M-04 uses `FrameTimingMetric` over a scripted navigation journey rather than `TraceSectionMetric`. The Navigation 3 `NavDisplay` setup does not expose a reliable span covering destination composition plus transition animation frames — tracing only the navigation event would measure the wrong work.

**Fixture ownership:** the `:macrobenchmark` module uses the benchmark-variant-only hook strategy — `BenchmarkSeedActivity` is exposed in `app/src/benchmark` for the `benchmark` build type only. It is not gated on `BuildConfig.DEBUG` (the benchmark variant is non-debuggable); it is gated on the `benchmark` build type.

**Threshold policy** (documented in `macrobenchmark/README.md`):
- Hard-fail on relative regression >20% against a committed 5-run baseline.
- Absolute NFR targets (e.g. M-01 P50 ≤ 2000ms) are advisory only — warn + artifact, never red-X CI.
- Swiftshader/emulator numbers are advisory regression signal, not authoritative. Authoritative perf requires a physical-device run.
- Nightly CI uses `swiftshader_indirect` (runner limitation); authoritative calibration requires `host` GPU mode.

## 4. Test infrastructure

### Test support helpers

| Helper | Location | Purpose |
|---|---|---|
| `RegressionHelpers.kt` | `testsupport/` | `killAndRelaunch`, `rotateMidEdit`, `seedVehicle`, `seedRecord`, `seedCustomCategory`, `seedAttachmentFile`, `seedCachedPremium`, `setBillingOverride`, vault path helpers, `NetworkState` |
| `VaultFixture.kt` | `testsupport/` | Vault fixture creation (local file or SAF), seeding, snapshot/diff comparison |
| `SystemPicker.kt` | `testsupport/` | UiAutomator-driven SAF folder picker and file picker helpers |
| `Camera.kt` | `testsupport/` | `captureAndAccept()`, `captureAndCancel()` via UiAutomator |
| `ExternalViewer.kt` | `testsupport/` | `hasResolver()` + `assertLaunchOrSkipUi()` (resolver-presence gate) |
| `FixtureAssetsRule.kt` | `testsupport/` | Copies bundled test assets to `MediaStore` Downloads with unique names; `@After` cleanup |
| `RetryRule.kt` | `testsupport/` | JUnit 4 retry rule with flake marker file generation |
| `ProcessDeath` | `testsupport/lifecycle/` | `.recreate()` (state-preservation via `scenario.recreate()`) and `.kill()` (hard kill via `am kill`) |
| `LauncherReentry` | `testsupport/lifecycle/` | `.launch()` — relaunch from launcher intent |
| `TrimMemory` | `testsupport/lifecycle/` | `.sendCritical()` — `am send-trim-memory` |
| `Gestures` | `testsupport/gestures/` | `.pinch()`, `.doubleTap()`, `.swipe()` for image viewer tests |
| `BillingSandboxEnvironment` | `testsupport/` | Multi-precondition `assumeReady()` skip guard + purchase flow helpers |

### Scaffold test

`regression/RegressionScaffoldTest.kt` — a minimal `@Regression` no-op test (`scaffoldIsDiscoverable() = Unit`) ensuring the regression package is discoverable by the AndroidJUnitRunner. Exists as an infrastructure sanity check from the initial setup.

### Quarantine policy

- Flake threshold: ≥1× in 20 nightly runs → quarantine.
- Core/PR tasks exclude quarantined tests via `notAnnotation=Quarantined`.
- Nightly includes quarantined tests (no `notAnnotation` filter).
- On quarantine: open a tracking issue with 14-day review.
- De-quarantine requires **30 consecutive green nightly runs** (authoritative source: [regression-promotion-playbook.md](regression-promotion-playbook.md)).

### Nightly dashboard & history

- `tools/aggregate_regression.py` runs after successful nightly/full regression runs.
- Produces [regression-dashboard.md](regression-dashboard.md) (suite health summary, quarantined tests, per-scenario status) and [regression-history.json](regression-history.json) (raw run data).
- Auto-committed by the `aggregate-results` job in `regression.yml`.

## 5. Billing sandbox device path

The existing GMD pool in `app/build.gradle.kts` is AOSP-only (`systemImageSource = "aosp"`). Real Google Play Billing requires a device with Google Play Services and the Play Store.

**Chosen approach: external device path.** Real billing verification runs outside GMD entirely via `billing-sandbox.yml` on a self-hosted runner with a real Play-enabled device. `regressionFullCheck` does not dispatch the sandbox invocation — the sandbox tier runs in its own dedicated workflow.

Key details:
- The workflow builds a `staging` variant signed with the same upload key as the Play internal track.
- Multi-precondition skip guard: `Assume.assumeTrue` checks `BILLING_SANDBOX_ACCOUNT`, signing material, device-path availability, signed-in Play account, and `BillingClient.queryProductDetailsAsync(premium_lifetime)` returning a non-empty list. Any missing prerequisite skips cleanly with a specific assumption message.
- `premium_lifetime` is non-consumable — the license-tester entitlement must be refunded/revoked in Play Console before each purchase run. Nightly scheduled runs execute R-P04 (restore) only; R-P03 (purchase) is manually triggered.
- See [billing-sandbox.md](billing-sandbox.md) for the full Play Console / CI runbook.