# Discrepancy Analysis: `plan-regression-tests.md` vs. Implemented Code > [!NOTE] > This document lists every discrepancy found between > [plan-regression-tests.md](file:///home/Messier82/projects/service-book/docs/plan-regression-tests.md) and the actual > implemented code. **Nothing has been fixed** โ€” this is an audit only. --- ## ๐Ÿ”ด Critical Discrepancies These are factually wrong statements in the doc that contradict reality. ### 1. Status banner says "No tests implemented yet" **Doc** (L3): > **Status:** Planning. No tests implemented yet โ€” see ยง4 Phased implementation. **Reality**: All phases are implemented โ€” 42+ test methods across 11 test classes, the macrobenchmark module, CI workflows, the dashboard, the promotion playbook, and the aggregation pipeline are all live and running. The dashboard records 4 historical runs. --- ### 2. `@Quarantined` is described as not yet carrying metadata parameters **Doc** (ยง4 Phase 7, L209): > Promote `@Quarantined` to carry `since: String` (ISO-8601), `issue: Int`, `reason: String` **Reality**: [Quarantined.kt](file:///home/Messier82/projects/service-book/app/src/androidTest/java/com/yorvana/testsupport/tiers/Quarantined.kt) already has all three parameters (`since`, `issue`, `reason`). This is implemented, not future work. --- ### 3. `@ScenarioId` is described as a future Phase 7 item **Doc** (ยง4 Phase 7, L211): > Adopt spec IDs โ€ฆ via `@ScenarioId("R-A01")` on the test method **Reality**: [ScenarioId.kt](file:///home/Messier82/projects/service-book/app/src/androidTest/java/com/yorvana/testsupport/tiers/ScenarioId.kt) exists and is already in use on several tests: - [LifecycleRobustnessRegressionTest.kt](file:///home/Messier82/projects/service-book/app/src/androidTest/java/com/yorvana/regression/lifecycle/LifecycleRobustnessRegressionTest.kt#L139) โ€” `@ScenarioId("R-L04")` - [VaultRealOsRegressionTest.kt](file:///home/Messier82/projects/service-book/app/src/androidTest/java/com/yorvana/regression/vault/VaultRealOsRegressionTest.kt#L89) โ€” `@ScenarioId("R-S02a")`, `@ScenarioId("R-S02b")`, `@ScenarioId("R-S02c")` - [BillingSandboxRegressionTest.kt](file:///home/Messier82/projects/service-book/app/src/androidTest/java/com/yorvana/regression/billing/BillingSandboxRegressionTest.kt#L88) โ€” `@ScenarioId("R-P04")` --- ### 4. Macrobenchmark module described as Phase 6 future work **Doc** (ยง4 Phase 6, L196โ€“205): > New `:macrobenchmark` module โ€ฆ M-01..04. Baseline thresholds calibrated over 5 runs **Reality**: The [macrobenchmark module](file:///home/Messier82/projects/service-book/macrobenchmark) exists and is fully wired: - [YorvanaBenchmark.kt](file:///home/Messier82/projects/service-book/macrobenchmark/src/main/java/com/yorvana/macrobenchmark/YorvanaBenchmark.kt) implements M-01 through M-04 - `regressionFullCheck` already [depends on](file:///home/Messier82/projects/service-book/app/build.gradle.kts#L397) `:macrobenchmark:pixel2api33BenchmarkAndroidTest` - The [regression-history.json](file:///home/Messier82/projects/service-book/docs/regression-history.json) records passing M-01..M-04 results - A [README.md](file:///home/Messier82/projects/service-book/macrobenchmark/README.md) documents the threshold policy --- ### 5. Phase 1 ยง4 describes items as not yet existing **Doc** (ยง4 Phase 1, L154โ€“163): Lists all Phase 1 deliverables as future work: - "Add custom marker annotations" โ†’ **All 6 exist** (`@Smoke`, `@Regression`, `@RegressionFull`, `@BillingSandbox`, `@Quarantined`, `@ScenarioId`) - "Wire `regressionCheck` and `regressionFullCheck` Gradle tasks" โ†’ **Both exist** using the GradleBuild child invocation mechanism - "Smoke isolation via positive filter" โ†’ **Implemented** at [app/build.gradle.kts L309-317](file:///home/Messier82/projects/service-book/app/build.gradle.kts#L309) - "Add UiAutomator helpers in `testsupport/`" โ†’ **All exist** (`SystemPicker`, `Camera`, `Permissions`, `ExternalViewer`) - "Add the annotation-misuse guardrail" โ†’ **Exists** as [RegressionTierScannerTest.kt](file:///home/Messier82/projects/service-book/app/src/test/java/com/yorvana/testsupport/RegressionTierScannerTest.kt) - "Add the nightly workflow pair" โ†’ **Both exist** (`regression.yml`, `regression-label.yml`) --- ### 6. All phases (2โ€“7) described as future work **Doc** (ยง4 Phase 2โ€“7, L165โ€“216): Every phase reads as future deliverables. **Reality**: All scenarios from Phases 1โ€“6 are implemented and running. Phase 7 items (quarantine metadata, `@ScenarioId`, orphan-quarantine guard, durable history, promotion playbook) are also either fully or substantially implemented. --- ## ๐ŸŸก Moderate Discrepancies These are areas where the doc's specifics differ from what was actually built. ### 7. R-D01 is in `setup/` not `vault/` โ€” plan says "ยง3.4 Vault / SAF" **Doc** (ยง3.4, L93): > R-D01 โ€” Fresh setup: drive real SAF folder picker via UiAutomator **Reality**: R-D01 lives in [SetupRegressionTest.kt](file:///home/Messier82/projects/service-book/app/src/androidTest/java/com/yorvana/regression/setup/SetupRegressionTest.kt) at `regression/setup/`, not `regression/vault/` or anywhere in the vault directory. The plan's ยง2 table suggests SAF picker tests live in `androidTest/.../regression/system/`. --- ### 8. Plan says "Phase 1 will pick one" mechanism โ€” but doesn't say which was chosen **Doc** (ยง2 L48โ€“50): > Two viable shapes; Phase 1 will pick one: > (a) `GradleBuild` child invocations โ€ฆ > (b) Variant- or device-scoped tasks โ€ฆ **Reality**: Option (a) โ€” `GradleBuild` child invocations โ€” was chosen, as seen in [registerRegressionDeviceTask](file:///home/Messier82/projects/service-book/app/build.gradle.kts#L325) which uses `tasks.register(taskName)`. The doc still presents both as open options. --- ### 9. R-D03 tests vault folder deletion, not permission revocation **Doc** (ยง3.4, L95): > R-D03 โ€” Vault permission revoked externally (clear persistable URI perms) โ†’ app shows recoverable error state **Reality**: [VaultRealOsRegressionTest.rD03](file:///home/Messier82/projects/service-book/app/src/androidTest/java/com/yorvana/regression/vault/VaultRealOsRegressionTest.kt#L63-L76) deletes the entire vault directory (`root.deleteRecursively()`) rather than revoking URI permissions. The test asserts "no crash on relaunch" but doesn't verify "recoverable error state" โ€” it just calls `killAndRelaunch()` with no assertion after. --- ### 10. R-D05 (Migration) is listed in ยง3.4 but doesn't exist **Doc** (ยง3.4, L97): > R-D05 โ€” Migration: pre-seed an old-format vault, verify `MigrationHelper` upgrades it **Reality**: No R-D05 test exists anywhere in the codebase. No `regression-fixtures/legacy-vault-v0/` directory exists. The plan itself notes this is conditional on a concrete legacy schema fixture being landed, but it's still listed as a scenario in ยง3.4 without being marked as unimplemented. --- ### 11. Dashboard lists `scaffoldIsDiscoverable` โ€” not in the plan's coverage map **Doc** (ยง3, L58โ€“150): The coverage map lists R-V01..07, R-R01..04, R-A01..07, R-D01..05, R-C01..03, R-S01..04, R-P01..05, R-L01..07, M-01..04. **Reality**: [RegressionScaffoldTest.scaffoldIsDiscoverable](file:///home/Messier82/projects/service-book/app/src/androidTest/java/com/yorvana/regression/RegressionScaffoldTest.kt) is a no-op test (`= Unit`) that exists and runs (present in the dashboard), but is never mentioned in the plan. It's clearly a Phase 1 infrastructure test but is not documented. --- ### 12. R-A03 tier discrepancy **Doc** (ยง3.3, L85): > R-A03 โ€” Image viewer gestures โ€ฆ **Owned by Phase 4 (#117)** The plan doesn't explicitly state the tier for R-A03, but Phase 4 discussion (L182) says: "long parametrized runs carry `@RegressionFull`, one canonical rotation + one canonical kill stay in `@Regression`". **Reality**: [ImageViewerGestureRegressionTest](file:///home/Messier82/projects/service-book/app/src/androidTest/java/com/yorvana/regression/attachments/ImageViewerGestureRegressionTest.kt#L35) has `@Regression` at class level, and the [dashboard](file:///home/Messier82/projects/service-book/docs/regression-dashboard.md#L31) lists R-A03 as `Regression` tier. This is consistent but the doc doesn't explicitly confirm the tier assignment for R-A03. --- ### 13. R-S02 three paths are in `VaultRealOsRegressionTest` not a settings test **Doc** (ยง3.6, L108): > R-S02 โ€” Change vault folder via Settings **Reality**: The three R-S02 tests (a/b/c) live in [VaultRealOsRegressionTest](file:///home/Messier82/projects/service-book/app/src/androidTest/java/com/yorvana/regression/vault/VaultRealOsRegressionTest.kt) alongside vault tests (R-D02..D04), not in a settings test class. The class is annotated `@RegressionFull`, so all three R-S02 variants run under RegressionFull tier. The plan specifies R-S02 as Phase 3 which is consistent with it living alongside other real-SAF scenarios, but the doc's ยง3.6 groups it under "Settings". --- ### 14. `regressionFullCheck` now includes macrobenchmark โ€” plan says Phase 6 will add it **Doc** (ยง2 L47): > the `:macrobenchmark` `dependsOn` is deferred to **Phase 6 (#119)** **Reality**: [regressionFullCheck](file:///home/Messier82/projects/service-book/app/build.gradle.kts#L394-L398) already `dependsOn(regressionFullCoreCheck, regressionFullExtendedCheck, macrobenchmarkTaskPath)`. The macrobenchmark dependency is live, not deferred. --- ### 15. Plan describes annotation-misuse guardrail implementation options โ€” only one was chosen **Doc** (ยง4 Phase 1, L162): > source-level scanner under `src/test` OR Gradle bytecode task scanning `compileDebugAndroidTestKotlin` output via ASM **Reality**: The source-level scanner approach was chosen โ€” implemented as a private object `RegressionTierScanner` inside [RegressionTierScannerTest.kt](file:///home/Messier82/projects/service-book/app/src/test/java/com/yorvana/testsupport/RegressionTierScannerTest.kt#L267). The doc still presents both options. --- ### 16. M-04 approach: plan says choose between TraceSectionMetric and FrameTimingMetric **Doc** (ยง3.9, L150): > Either `TraceSectionMetric` against production `androidx.tracing` sections โ€ฆ **or** a `FrameTimingMetric` **Reality**: [YorvanaBenchmark.M04](file:///home/Messier82/projects/service-book/macrobenchmark/src/main/java/com/yorvana/macrobenchmark/YorvanaBenchmark.kt#L83) uses `FrameTimingMetric`. The [README](file:///home/Messier82/projects/service-book/macrobenchmark/README.md#L59-L62) explains the choice was made (no reliable transition-work span). The plan still presents both options. --- ### 17. R-L06 plan specifies two backends (file-backed + SAF) โ€” only file-backed is tested **Doc** (ยง3.8, L138โ€“142): > Assertion split by vault backend โ€ฆ **File-backed** โ€ฆ **SAF** โ€ฆ **Reality**: [StorageRobustnessRegressionTest](file:///home/Messier82/projects/service-book/app/src/androidTest/java/com/yorvana/regression/vault/StorageRobustnessRegressionTest.kt) tests only the file-backed path via `VaultStorageImpl` with `Uri.fromFile(root)`. No SAF-backed R-L06 test exists. The plan acknowledges this may narrow ("If either production change is descoped"), but the doc doesn't record which was chosen. --- ### 18. Dashboard lists M-01..M-04 as `Regression` tier **Dashboard** ([regression-dashboard.md L25-28](file:///home/Messier82/projects/service-book/docs/regression-dashboard.md#L25)): > M-01 through M-04 all listed as tier `Regression` **Plan** (ยง2, L42): > | Macrobenchmark | separate module | Nightly | ~15 min | The plan says macrobenchmarks are a "separate module" tier running nightly. The dashboard labels them `Regression`. Since macrobenchmarks live in their own module and don't carry any app-tier annotation, labeling them `Regression` is misleading โ€” they're not filtered by `annotation=Regression` in CI. --- ### 19. Fixture ownership approach for macrobenchmark โ€” plan listed three options **Doc** (ยง4 Phase 6, L203): > (a) extract seeding into a shared `:testfixtures` module, > (b) expose a benchmark-variant-only test hook โ€ฆ > (c) intentional duplication with sync requirement **Reality**: Option (b) was chosen โ€” `BenchmarkSeedActivity` is exposed only in the `benchmark` variant per the [README](file:///home/Messier82/projects/service-book/macrobenchmark/README.md#L26-L29). The plan doesn't record the decision. --- ### 20. `regressionFullSandboxCheck` described in ยง2 but doesn't exist in Gradle **Doc** (ยง2, L54): > `regressionFullSandboxCheck` โ€” runs `@BillingSandbox` against the Play-enabled device path **Reality**: No `regressionFullSandboxCheck` Gradle task exists. The plan itself later notes (L190) that "Option 2 selected โ€” external `billing-sandbox.yml` workflow โ€ฆ with the sandbox tier outside `regressionFullCheck`." But ยง2 still describes the task as if it could exist. This should be cleaned up. --- ### 21. `preserveRegressionCoreResults` task not documented anywhere in the plan **Reality**: [preserveRegressionCoreResults](file:///home/Messier82/projects/service-book/app/build.gradle.kts#L371-L389) is a `Copy` task that saves `@Regression` XML results before `regressionFullExtendedCheck` overwrites the shared `managedDevice/` output directory. This is a necessary implementation detail but is nowhere in the plan. --- ## ๐ŸŸข Minor / Cosmetic Discrepancies ### 22. Plan references line numbers that may have drifted **Doc** (L26): `app/build.gradle.kts:68` for `testBuildType`, (L43) for `testVariant`. These line references are brittle and likely stale after code changes. ### 23. Plan uses `categories/` as a subpackage โ€” code uses `settings/` **Doc** (ยง3.5): Categories tests (R-C01..03) have their own section. In the code, they live inside [SettingsCategoriesPaywallRegressionTest.kt](file:///home/Messier82/projects/service-book/app/src/androidTest/java/com/yorvana/regression/settings/SettingsCategoriesPaywallRegressionTest.kt) in the `settings/` package, alongside R-S01, R-S03, R-S04, and R-P05. ### 24. Plan says `testsupport/lifecycle/` for lifecycle helpers **Doc** (ยง4 Phase 4, L184): > Helpers under `testsupport/lifecycle/`: `ProcessDeath.recreate` โ€ฆ and `ProcessDeath.kill` **Reality**: `ProcessDeath`, `LauncherReentry`, and `TrimMemory` do exist under `testsupport/lifecycle/` (confirmed by imports in LifecycleRobustnessRegressionTest). โœ… This matches. ### 25. Plan says `testsupport/gestures/` for gesture helpers **Doc** (ยง4 Phase 4, L184): > Gesture helpers under `testsupport/gestures/`: `Gestures.pinch / .doubleTap / .swipe` **Reality**: Confirmed by imports โ€” `Gestures.pinch`, `Gestures.doubleTap`, `Gestures.swipe`, `Direction` all exist under `testsupport/gestures/`. โœ… This matches. ### 26. Plan references issue numbers (#114โ€“#120) as future contracts Throughout the doc, phrases like "issue **#114**", "Phase 5 (#118)" are used to point to GitHub issues. These references are correct as historical context but read as if the work is pending. --- ## โœ… What Matches Correctly For completeness, these plan claims were verified as accurate: | Claim | Status | |---|---| | 5 + 1 tier annotations with `@Inherited` and `RUNTIME` retention | โœ… All 6 match exactly | | `GradleBuild` child invocation mechanism for regression tasks | โœ… | | `regressionCheck` = `annotation=Regression, notAnnotation=Quarantined` | โœ… | | `regressionFullCoreCheck` = `annotation=Regression` (no notAnnotation) | โœ… | | `regressionFullExtendedCheck` = `annotation=RegressionFull` | โœ… | | `SmokeTest.kt` carries `@Smoke` | โœ… | | Smoke device uses positive `annotation=Smoke` filter | โœ… | | `regression.yml` runs `regressionCheck` on non-nightly, `regressionFullCheck` on nightly | โœ… | | `regression-label.yml` triggers on `regression` label | โœ… | | `billing-sandbox.yml` runs on self-hosted runner with physical device | โœ… | | Billing sandbox uses Option 2 (external workflow) | โœ… | | R-V01..07 all implemented and `@Regression` tier | โœ… | | R-R01..04 all implemented and `@Regression` tier | โœ… | | R-C01..03 implemented | โœ… | | R-S01, R-S03, R-S04 implemented | โœ… | | R-P05 implemented as `@Regression` (not sandbox-dependent) | โœ… | | R-L01..L05, R-L07 implemented | โœ… | | R-L01 split: canonical `@Regression` + extended `@RegressionFull` | โœ… | | R-L02 uses `ProcessDeath.kill` (not `force-stop`) | โœ… | | R-A01..A02, R-A04..A07 implemented as `@RegressionFull` | โœ… | | R-A03 implemented as `@Regression` | โœ… | | R-D01..D04 implemented | โœ… | | R-S02a/b/c implemented as `@RegressionFull` | โœ… | | R-P03, R-P04 implemented as `@BillingSandbox` | โœ… | | R-P01, R-P02 intentionally not reimplemented (covered by smoke) | โœ… | | M-01..M-04 implemented in `:macrobenchmark` module | โœ… | | Dashboard + history JSON + promotion playbook exist | โœ… | | Aggregate pipeline (`tools/aggregate_regression.py`) exists | โœ… | | No `@Quarantined` tests in production (consistent with dashboard) | โœ… | | Annotation-misuse scanner with negative test fixtures | โœ… | --- ## Summary | Category | Count | |---|---| | ๐Ÿ”ด Critical (doc says "not implemented" but it is) | 6 | | ๐ŸŸก Moderate (specifics differ from implementation) | 15 | | ๐ŸŸข Minor / Cosmetic | 5 | | โœ… Verified correct | 35+ | The primary theme: **the doc is frozen in a "planning" state and needs to be rewritten as "implemented documentation."** All the "Phase X willโ€ฆ" language, the "Planning. No tests implemented yet" status banner, and the open decision points need to be resolved to reflect the choices that were actually made.