257 lines
19 KiB
Markdown
257 lines
19 KiB
Markdown
# Tasks: Test Suite Governance & Performance Foundation
|
|
|
|
**Input**: Design documents from `/specs/206-test-suite-governance/`
|
|
**Prerequisites**: plan.md (required), spec.md (required for user stories), research.md, data-model.md, contracts/, quickstart.md
|
|
|
|
**Tests**: Tests are REQUIRED for this feature because it changes checked-in test execution paths, shared fixture behavior, budget reporting, and guard coverage in a Laravel/Pest codebase.
|
|
**Operations / RBAC / UI surfaces**: Not applicable to product runtime. This feature is limited to repository test infrastructure, checked-in commands, reporting artifacts, and test-authoring discipline.
|
|
|
|
## Phase 1: Setup (Shared Infrastructure)
|
|
|
|
**Purpose**: Establish the checked-in lane manifest and reporting scaffolding shared by all user stories.
|
|
|
|
- [X] T001 Create the checked-in lane manifest and artifact-path skeleton in `apps/platform/tests/Support/TestLaneManifest.php` and `apps/platform/storage/logs/test-lanes/.gitignore`
|
|
- [X] T002 [P] Create reusable lane budget and report helpers in `apps/platform/tests/Support/TestLaneBudget.php` and `apps/platform/tests/Support/TestLaneReport.php`
|
|
- [X] T003 [P] Create the base lane guard and support test skeletons in `apps/platform/tests/Feature/Guards/TestLaneManifestTest.php`, `apps/platform/tests/Feature/Guards/TestLaneCommandContractTest.php`, and `apps/platform/tests/Unit/Support/TestLaneBudgetTest.php`
|
|
|
|
---
|
|
|
|
## Phase 2: Foundational (Blocking Prerequisites)
|
|
|
|
**Purpose**: Put the reusable lane, artifact, and command-routing seams in place before any story-specific lane work begins.
|
|
|
|
**⚠️ CRITICAL**: No user story work can begin until this phase is complete.
|
|
|
|
- [X] T004 Implement the shared lane selector, artifact, and budget model in `apps/platform/tests/Support/TestLaneManifest.php`, `apps/platform/tests/Support/TestLaneBudget.php`, and `apps/platform/tests/Support/TestLaneReport.php`
|
|
- [X] T005 [P] Add the checked-in Sail-friendly lane runner and reporting scripts in `scripts/platform-test-lane` and `scripts/platform-test-report`
|
|
- [X] T006 [P] Wire lane-facing command aliases and app-root artifact targets in `apps/platform/composer.json`, `apps/platform/phpunit.xml`, and `apps/platform/phpunit.pgsql.xml`
|
|
- [X] T007 Add foundational guard coverage for manifest validity, command routing, and artifact path behavior in `apps/platform/tests/Feature/Guards/TestLaneManifestTest.php`, `apps/platform/tests/Feature/Guards/TestLaneCommandContractTest.php`, and `apps/platform/tests/Unit/Support/TestLaneBudgetTest.php`
|
|
|
|
**Checkpoint**: Lane declarations, runner/report scripts, and guard scaffolding exist so story work can proceed independently.
|
|
|
|
---
|
|
|
|
## Phase 3: User Story 1 - Run The Fast Feedback Lane (Priority: P1) 🎯 MVP
|
|
|
|
**Goal**: Give contributors one checked-in default run that is clearly faster than the broad suite and that excludes browser and intentionally heavy governance families.
|
|
|
|
**Independent Test**: Run the default checked-in lane command from a clean environment and verify that it resolves to the fast-feedback selectors, runs in parallel, excludes browser and heavy governance families, and stays within the documented fast-lane budget target.
|
|
|
|
### Tests for User Story 1
|
|
|
|
- [X] T008 [P] [US1] Add fast-feedback default-run contract coverage in `apps/platform/tests/Feature/Guards/FastFeedbackLaneContractTest.php` and `apps/platform/tests/Feature/Guards/TestLaneCommandContractTest.php`
|
|
- [X] T009 [P] [US1] Add browser and heavy-family exclusion coverage for the default lane in `apps/platform/tests/Feature/Guards/FastFeedbackLaneExclusionTest.php` and `apps/platform/tests/Feature/Guards/TestLaneManifestTest.php`
|
|
|
|
### Implementation for User Story 1
|
|
|
|
- [X] T010 [US1] Define the `fast-feedback` lane selectors, purpose metadata, intended audience, included or excluded families, ownership expectations, parallel mode, and default-entry semantics in `apps/platform/tests/Support/TestLaneManifest.php`
|
|
- [X] T011 [US1] Wire the fast-feedback checked-in commands and default test alias behavior in `apps/platform/composer.json` and `scripts/platform-test-lane`
|
|
- [X] T012 [US1] Encode the initial fast-lane exclusions, intended audience, ownership expectations, and contributor escalation guidance in `apps/platform/tests/Support/TestLaneManifest.php` and `README.md`
|
|
|
|
**Checkpoint**: The repository has a single fast default lane that is independently runnable and excludes browser plus intentionally heavy governance families.
|
|
|
|
---
|
|
|
|
## Phase 4: User Story 2 - Run The Broader Confidence Lane Before Merge (Priority: P1)
|
|
|
|
**Goal**: Provide a broader pre-merge lane plus separate browser and heavy-governance paths without collapsing everything back into the old full-suite default.
|
|
|
|
**Independent Test**: Run the confidence, browser, and heavy-governance commands independently and verify that confidence remains broader than fast-feedback, stays below the current full-suite baseline, browser stays isolated, and the heavy-governance lane owns the first intentionally expensive families.
|
|
|
|
### Tests for User Story 2
|
|
|
|
- [X] T013 [P] [US2] Add confidence-lane inclusion, manifest-scope, below-full-suite-baseline, and heavy-lane separation coverage in `apps/platform/tests/Feature/Guards/ConfidenceLaneContractTest.php` and `apps/platform/tests/Feature/Guards/HeavyGovernanceLaneContractTest.php`
|
|
- [X] T014 [P] [US2] Add browser-lane isolation and command-routing coverage in `apps/platform/tests/Feature/Guards/BrowserLaneIsolationTest.php` and `apps/platform/tests/Feature/Guards/TestLaneCommandContractTest.php`
|
|
|
|
### Implementation for User Story 2
|
|
|
|
- [X] T015 [US2] Define the `confidence` lane as the Unit suite plus the manifest-defined non-browser Feature or Integration selectors, and seed the `heavy-governance` plus `browser` selectors with explicit purpose, audience, included or excluded families, and ownership metadata in `apps/platform/tests/Support/TestLaneManifest.php`
|
|
- [X] T016 [US2] Wire the checked-in confidence, browser, and heavy-governance commands in `apps/platform/composer.json` and `scripts/platform-test-lane`
|
|
- [X] T017 [US2] Document the confidence-versus-browser-versus-heavy usage model, intended audience, and ownership expectations in `README.md` and `specs/206-test-suite-governance/quickstart.md`
|
|
|
|
**Checkpoint**: Contributors and reviewers can independently run a broader confidence lane, a browser lane, and a seed heavy-governance lane without ambiguity.
|
|
|
|
---
|
|
|
|
## Phase 5: User Story 3 - See Where Runtime Is Getting Worse (Priority: P2)
|
|
|
|
**Goal**: Make slow-test drift visible through checked-in profiling and machine-readable reporting paths, plus initial documented runtime budgets.
|
|
|
|
**Independent Test**: Run the reporting and profiling commands and verify that they emit app-root-relative artifacts, show at least the top 10 slowest entries, and evaluate the run against documented lane budgets plus family thresholds.
|
|
|
|
### Tests for User Story 3
|
|
|
|
- [X] T018 [P] [US3] Add at-least-top-10 JUnit, summary-artifact, family-budget, and report-shape coverage in `apps/platform/tests/Unit/Support/TestLaneReportTest.php` and `apps/platform/tests/Feature/Guards/TestLaneArtifactsContractTest.php`
|
|
- [X] T019 [P] [US3] Add profiling-mode, baseline-capture, and budget-status coverage in `apps/platform/tests/Feature/Guards/ProfileLaneContractTest.php` and `apps/platform/tests/Unit/Support/TestLaneBudgetTest.php`
|
|
|
|
### Implementation for User Story 3
|
|
|
|
- [X] T020 [US3] Implement JUnit, summary, and budget artifact generation in `scripts/platform-test-report`, `apps/platform/tests/Support/TestLaneReport.php`, and `apps/platform/storage/logs/test-lanes/.gitignore`
|
|
- [X] T021 [US3] Add serial profiling and JUnit command paths in `apps/platform/composer.json` and `scripts/platform-test-lane`
|
|
- [X] T022 [US3] Capture the current full-suite baseline and the first measured lane baselines, including explicit verification that `fast-feedback` is at least 50% below the current full-suite baseline and `confidence` remains below the current full-suite baseline, in `README.md` and `specs/206-test-suite-governance/quickstart.md`
|
|
- [X] T023 [US3] Refine the heavy-governance selectors from profiling evidence and publish the initial lane budgets plus heavy-family thresholds in `apps/platform/tests/Support/TestLaneManifest.php`, `README.md`, and `specs/206-test-suite-governance/quickstart.md`
|
|
|
|
**Checkpoint**: The repository can emit machine-readable test artifacts, list at least the top 10 slowest entries, and compare runs against baseline-backed lane budgets plus family thresholds.
|
|
|
|
---
|
|
|
|
## Phase 6: User Story 4 - Author Cheap, Honest Tests By Default (Priority: P2)
|
|
|
|
**Goal**: Stop the most common shared helper and one additional factory cluster from silently provisioning heavy tenant/provider graphs unless the test explicitly asks for them.
|
|
|
|
**Independent Test**: Use the shared helper and targeted factory states in isolation, verify that the default path stays minimal, and confirm that representative high-usage callers can opt into heavier behavior explicitly where needed.
|
|
|
|
### Tests for User Story 4
|
|
|
|
- [X] T024 [P] [US4] Add shared-helper profile coverage for provider, credential, session, cache, workspace, membership, and UI-context opt-ins in `apps/platform/tests/Unit/Support/CreateUserWithTenantProfilesTest.php` and `apps/platform/tests/Feature/Guards/FixtureCostProfilesGuardTest.php`
|
|
- [X] T025 [P] [US4] Add minimal-versus-heavy factory state coverage in `apps/platform/tests/Unit/Factories/TenantFactoryTest.php` and `apps/platform/tests/Unit/Factories/ProviderConnectionFactoryTest.php`
|
|
- [X] T026 [P] [US4] Add taxonomy-audit and lane-placement guard coverage in `apps/platform/tests/Feature/Guards/TestTaxonomyPlacementGuardTest.php` and `apps/platform/tests/Feature/Guards/ConfidenceLaneContractTest.php`
|
|
|
|
### Implementation for User Story 4
|
|
|
|
- [X] T027 [US4] Split `createUserWithTenant()` into minimal and explicit heavy, provider-enabled, credential-enabled, or UI-context profiles in `apps/platform/tests/Pest.php`
|
|
- [X] T028 [US4] Introduce explicit minimal and heavy state behavior for hidden workspace or provider graph creation in `apps/platform/database/factories/TenantFactory.php` and `apps/platform/database/factories/ProviderConnectionFactory.php`
|
|
- [X] T029 [US4] Audit and reclassify the first obvious heavy-governance batch in `apps/platform/tests/Support/TestLaneManifest.php`, covering `apps/platform/tests/Architecture/PlatformVocabularyBoundaryGuardTest.php`, `apps/platform/tests/Architecture/ReasonTranslationPrimarySurfaceGuardTest.php`, `apps/platform/tests/Deprecation/IsPlatformSuperadminDeprecationTest.php`, and `apps/platform/tests/Feature/Guards/ActionSurfaceContractTest.php`
|
|
- [X] T030 [US4] Migrate the first high-usage callers to the new fixture profiles in `apps/platform/tests/Feature/Guards/ActionSurfaceContractTest.php`, `apps/platform/tests/Feature/OpsUx/OperateHubShellTest.php`, `apps/platform/tests/Feature/Findings/FindingExceptionRegisterTest.php`, and `apps/platform/tests/Feature/EntraAdminRoles/EntraAdminRolesFindingGeneratorTest.php`
|
|
- [X] T031 [US4] Document honest taxonomy, DB reset rules, seed restrictions, and schema-baseline follow-up guidance in `README.md` and `specs/206-test-suite-governance/quickstart.md`
|
|
|
|
**Checkpoint**: Shared test setup defaults are materially cheaper, and the first obvious taxonomy misfits have been reclassified or regrouped with written guidance for future tests.
|
|
|
|
---
|
|
|
|
## Phase 7: Polish & Cross-Cutting Concerns
|
|
|
|
**Purpose**: Reconcile the implementation with the design artifacts and validate the end-to-end workflow.
|
|
|
|
- [X] T032 [P] Reconcile the implemented lane model with `specs/206-test-suite-governance/contracts/test-lane-manifest.schema.json`, `specs/206-test-suite-governance/contracts/test-suite-governance.logical.openapi.yaml`, and `specs/206-test-suite-governance/quickstart.md`
|
|
- [X] T033 Run the focused validation flow documented in `specs/206-test-suite-governance/quickstart.md`
|
|
- [X] T034 Run formatting with `cd apps/platform && ./vendor/bin/sail bin pint --dirty --format agent`
|
|
|
|
---
|
|
|
|
## Dependencies & Execution Order
|
|
|
|
### Phase Dependencies
|
|
|
|
- **Setup (Phase 1)**: No dependencies, can start immediately.
|
|
- **Foundational (Phase 2)**: Depends on Setup completion and blocks all story work.
|
|
- **User Story 1 (Phase 3)**: Starts after Foundational completion.
|
|
- **User Story 2 (Phase 4)**: Starts after Foundational completion and can proceed in parallel with US1.
|
|
- **User Story 3 (Phase 5)**: Starts after Foundational completion; it benefits from US1 and US2 landing first so the lane inventory is stable.
|
|
- **User Story 4 (Phase 6)**: Starts after Foundational completion; it can proceed in parallel with US3 once the lane manifest and command seams exist.
|
|
- **Polish (Phase 7)**: Runs after the desired user stories are complete.
|
|
|
|
### User Story Dependencies
|
|
|
|
- **User Story 1 (P1)**: No dependency on other stories. This is the recommended MVP slice.
|
|
- **User Story 2 (P1)**: Depends only on the foundational lane and command scaffolding, not on US1 completion.
|
|
- **User Story 3 (P2)**: Depends on the foundational lane and artifact scaffolding; it is strongest once US1 and US2 stabilize the lane inventory.
|
|
- **User Story 4 (P2)**: Depends on the foundational lane/guard scaffolding; it does not require US3 but shares docs and reporting outcomes with it.
|
|
|
|
### Within Each User Story
|
|
|
|
- Tests MUST be written and fail before implementation.
|
|
- Lane manifest and command-routing seams must exist before story-specific lane work begins.
|
|
- Artifact generation, baseline capture, and profiling refinement must exist before budget publication is considered complete.
|
|
- Helper and factory profile changes must land before high-usage caller migrations are considered complete.
|
|
- The first taxonomy audit must land before the initial reclassification batch is considered complete.
|
|
|
|
### Parallel Opportunities
|
|
|
|
- T002 and T003 can run in parallel.
|
|
- T005 and T006 can run in parallel.
|
|
- US1 test tasks T008 and T009 can run in parallel.
|
|
- US2 test tasks T013 and T014 can run in parallel.
|
|
- US3 test tasks T018 and T019 can run in parallel.
|
|
- US4 test tasks T024, T025, and T026 can run in parallel.
|
|
- After Foundational completion, US2, US3, and US4 can proceed in parallel with separate owners.
|
|
|
|
---
|
|
|
|
## Parallel Example: User Story 1
|
|
|
|
```bash
|
|
# Launch the fast-feedback guard coverage together:
|
|
Task: "Add fast-feedback default-run contract coverage in apps/platform/tests/Feature/Guards/FastFeedbackLaneContractTest.php and apps/platform/tests/Feature/Guards/TestLaneCommandContractTest.php"
|
|
Task: "Add browser and heavy-family exclusion coverage for the default lane in apps/platform/tests/Feature/Guards/FastFeedbackLaneExclusionTest.php and apps/platform/tests/Feature/Guards/TestLaneManifestTest.php"
|
|
|
|
# Then land the fast lane implementation:
|
|
Task: "Define the fast-feedback lane selectors, parallel mode, and default-entry semantics in apps/platform/tests/Support/TestLaneManifest.php"
|
|
Task: "Wire the fast-feedback checked-in commands and default test alias behavior in apps/platform/composer.json and scripts/platform-test-lane"
|
|
```
|
|
|
|
---
|
|
|
|
## Parallel Example: User Story 2
|
|
|
|
```bash
|
|
# Launch the broader-lane guard coverage together:
|
|
Task: "Add confidence-lane inclusion and heavy-lane separation coverage in apps/platform/tests/Feature/Guards/ConfidenceLaneContractTest.php and apps/platform/tests/Feature/Guards/HeavyGovernanceLaneContractTest.php"
|
|
Task: "Add browser-lane isolation and command-routing coverage in apps/platform/tests/Feature/Guards/BrowserLaneIsolationTest.php and apps/platform/tests/Feature/Guards/TestLaneCommandContractTest.php"
|
|
|
|
# Then land the lane separation work:
|
|
Task: "Define the confidence, heavy-governance, and browser lane selectors with baseline-backed budget metadata in apps/platform/tests/Support/TestLaneManifest.php"
|
|
Task: "Wire the checked-in confidence, browser, and heavy-governance commands in apps/platform/composer.json and scripts/platform-test-lane"
|
|
```
|
|
|
|
---
|
|
|
|
## Parallel Example: User Story 3
|
|
|
|
```bash
|
|
# Launch the reporting coverage together:
|
|
Task: "Add at-least-top-10 JUnit, summary-artifact, family-budget, and report-shape coverage in apps/platform/tests/Unit/Support/TestLaneReportTest.php and apps/platform/tests/Feature/Guards/TestLaneArtifactsContractTest.php"
|
|
Task: "Add profiling-mode, baseline-capture, and budget-status coverage in apps/platform/tests/Feature/Guards/ProfileLaneContractTest.php and apps/platform/tests/Unit/Support/TestLaneBudgetTest.php"
|
|
|
|
# Then land the report generation and baseline work:
|
|
Task: "Implement JUnit, summary, and budget artifact generation in scripts/platform-test-report, apps/platform/tests/Support/TestLaneReport.php, and apps/platform/storage/logs/test-lanes/.gitignore"
|
|
Task: "Capture the current full-suite baseline and the first measured lane baselines in README.md and specs/206-test-suite-governance/quickstart.md"
|
|
```
|
|
|
|
---
|
|
|
|
## Parallel Example: User Story 4
|
|
|
|
```bash
|
|
# Launch the cheap-default coverage together:
|
|
Task: "Add shared-helper profile coverage in apps/platform/tests/Unit/Support/CreateUserWithTenantProfilesTest.php and apps/platform/tests/Feature/Guards/FixtureCostProfilesGuardTest.php"
|
|
Task: "Add minimal-versus-heavy factory state coverage in apps/platform/tests/Unit/Factories/TenantFactoryTest.php and apps/platform/tests/Unit/Factories/ProviderConnectionFactoryTest.php"
|
|
Task: "Add taxonomy-audit and lane-placement guard coverage in apps/platform/tests/Feature/Guards/TestTaxonomyPlacementGuardTest.php and apps/platform/tests/Feature/Guards/ConfidenceLaneContractTest.php"
|
|
|
|
# Then land the fixture changes and taxonomy work:
|
|
Task: "Split createUserWithTenant() into minimal and explicit heavy or provider-enabled profiles in apps/platform/tests/Pest.php"
|
|
Task: "Audit and reclassify the first obvious heavy-governance batch in apps/platform/tests/Support/TestLaneManifest.php, covering apps/platform/tests/Architecture/PlatformVocabularyBoundaryGuardTest.php, apps/platform/tests/Architecture/ReasonTranslationPrimarySurfaceGuardTest.php, apps/platform/tests/Deprecation/IsPlatformSuperadminDeprecationTest.php, and apps/platform/tests/Feature/Guards/ActionSurfaceContractTest.php"
|
|
```
|
|
|
|
---
|
|
|
|
## Implementation Strategy
|
|
|
|
### MVP First (User Story 1 Only)
|
|
|
|
1. Complete Phase 1: Setup.
|
|
2. Complete Phase 2: Foundational.
|
|
3. Complete Phase 3: User Story 1.
|
|
4. Validate that the default contributor path now resolves to the fast-feedback lane and excludes browser plus heavy-governance families.
|
|
|
|
### Incremental Delivery
|
|
|
|
1. Land the shared lane manifest, runner, and report scaffolding.
|
|
2. Ship the fast-feedback lane as the first visible contributor improvement.
|
|
3. Add the broader confidence lane plus separate browser and heavy-governance paths.
|
|
4. Add reporting, budgets, and profiling visibility.
|
|
5. Finish with cheap shared defaults and taxonomy guidance so new tests stop reintroducing hidden cost.
|
|
|
|
### Parallel Team Strategy
|
|
|
|
1. One developer lands the foundational manifest and runner/report scaffolding.
|
|
2. A second developer can own fast-feedback and confidence lane separation while another owns reporting and budget output.
|
|
3. A final developer can drive the helper/factory cheap-default work and migrate the highest-usage caller files once the new fixture profiles exist.
|
|
|
|
## Notes
|
|
|
|
- `[P]` tasks are limited to work on different files with no incomplete dependency overlap.
|
|
- US1 is the recommended MVP because it restores the default contributor loop first.
|
|
- US2 turns the lane model into a full pre-merge workflow by separating confidence, browser, and heavy-governance paths.
|
|
- US3 makes slow-test regressions visible and measurable.
|
|
- US4 ensures the suite does not grow back into hidden heavy defaults after the lane model lands. |