Ahmed Darrazi ba6ac78ad0 test: implement suite governance foundation

2026-04-16 15:57:39 +02:00

19 KiB

Raw Blame History

Tasks: Test Suite Governance & Performance Foundation

Input: Design documents from /specs/206-test-suite-governance/ Prerequisites: plan.md (required), spec.md (required for user stories), research.md, data-model.md, contracts/, quickstart.md

Tests: Tests are REQUIRED for this feature because it changes checked-in test execution paths, shared fixture behavior, budget reporting, and guard coverage in a Laravel/Pest codebase. Operations / RBAC / UI surfaces: Not applicable to product runtime. This feature is limited to repository test infrastructure, checked-in commands, reporting artifacts, and test-authoring discipline.

Phase 1: Setup (Shared Infrastructure)

Purpose: Establish the checked-in lane manifest and reporting scaffolding shared by all user stories.

T001 Create the checked-in lane manifest and artifact-path skeleton in apps/platform/tests/Support/TestLaneManifest.php and apps/platform/storage/logs/test-lanes/.gitignore
T002 [P] Create reusable lane budget and report helpers in apps/platform/tests/Support/TestLaneBudget.php and apps/platform/tests/Support/TestLaneReport.php
T003 [P] Create the base lane guard and support test skeletons in apps/platform/tests/Feature/Guards/TestLaneManifestTest.php, apps/platform/tests/Feature/Guards/TestLaneCommandContractTest.php, and apps/platform/tests/Unit/Support/TestLaneBudgetTest.php

Phase 2: Foundational (Blocking Prerequisites)

Purpose: Put the reusable lane, artifact, and command-routing seams in place before any story-specific lane work begins.

⚠️ CRITICAL: No user story work can begin until this phase is complete.

T004 Implement the shared lane selector, artifact, and budget model in apps/platform/tests/Support/TestLaneManifest.php, apps/platform/tests/Support/TestLaneBudget.php, and apps/platform/tests/Support/TestLaneReport.php
T005 [P] Add the checked-in Sail-friendly lane runner and reporting scripts in scripts/platform-test-lane and scripts/platform-test-report
T006 [P] Wire lane-facing command aliases and app-root artifact targets in apps/platform/composer.json, apps/platform/phpunit.xml, and apps/platform/phpunit.pgsql.xml
T007 Add foundational guard coverage for manifest validity, command routing, and artifact path behavior in apps/platform/tests/Feature/Guards/TestLaneManifestTest.php, apps/platform/tests/Feature/Guards/TestLaneCommandContractTest.php, and apps/platform/tests/Unit/Support/TestLaneBudgetTest.php

Checkpoint: Lane declarations, runner/report scripts, and guard scaffolding exist so story work can proceed independently.

Phase 3: User Story 1 - Run The Fast Feedback Lane (Priority: P1) 🎯 MVP

Goal: Give contributors one checked-in default run that is clearly faster than the broad suite and that excludes browser and intentionally heavy governance families.

Independent Test: Run the default checked-in lane command from a clean environment and verify that it resolves to the fast-feedback selectors, runs in parallel, excludes browser and heavy governance families, and stays within the documented fast-lane budget target.

Tests for User Story 1

T008 [P] [US1] Add fast-feedback default-run contract coverage in apps/platform/tests/Feature/Guards/FastFeedbackLaneContractTest.php and apps/platform/tests/Feature/Guards/TestLaneCommandContractTest.php
T009 [P] [US1] Add browser and heavy-family exclusion coverage for the default lane in apps/platform/tests/Feature/Guards/FastFeedbackLaneExclusionTest.php and apps/platform/tests/Feature/Guards/TestLaneManifestTest.php

Implementation for User Story 1

T010 [US1] Define the fast-feedback lane selectors, purpose metadata, intended audience, included or excluded families, ownership expectations, parallel mode, and default-entry semantics in apps/platform/tests/Support/TestLaneManifest.php
T011 [US1] Wire the fast-feedback checked-in commands and default test alias behavior in apps/platform/composer.json and scripts/platform-test-lane
T012 [US1] Encode the initial fast-lane exclusions, intended audience, ownership expectations, and contributor escalation guidance in apps/platform/tests/Support/TestLaneManifest.php and README.md

Checkpoint: The repository has a single fast default lane that is independently runnable and excludes browser plus intentionally heavy governance families.

Phase 4: User Story 2 - Run The Broader Confidence Lane Before Merge (Priority: P1)

Goal: Provide a broader pre-merge lane plus separate browser and heavy-governance paths without collapsing everything back into the old full-suite default.

Independent Test: Run the confidence, browser, and heavy-governance commands independently and verify that confidence remains broader than fast-feedback, stays below the current full-suite baseline, browser stays isolated, and the heavy-governance lane owns the first intentionally expensive families.

Tests for User Story 2

T013 [P] [US2] Add confidence-lane inclusion, manifest-scope, below-full-suite-baseline, and heavy-lane separation coverage in apps/platform/tests/Feature/Guards/ConfidenceLaneContractTest.php and apps/platform/tests/Feature/Guards/HeavyGovernanceLaneContractTest.php
T014 [P] [US2] Add browser-lane isolation and command-routing coverage in apps/platform/tests/Feature/Guards/BrowserLaneIsolationTest.php and apps/platform/tests/Feature/Guards/TestLaneCommandContractTest.php

Implementation for User Story 2

T015 [US2] Define the confidence lane as the Unit suite plus the manifest-defined non-browser Feature or Integration selectors, and seed the heavy-governance plus browser selectors with explicit purpose, audience, included or excluded families, and ownership metadata in apps/platform/tests/Support/TestLaneManifest.php
T016 [US2] Wire the checked-in confidence, browser, and heavy-governance commands in apps/platform/composer.json and scripts/platform-test-lane
T017 [US2] Document the confidence-versus-browser-versus-heavy usage model, intended audience, and ownership expectations in README.md and specs/206-test-suite-governance/quickstart.md

Checkpoint: Contributors and reviewers can independently run a broader confidence lane, a browser lane, and a seed heavy-governance lane without ambiguity.

Phase 5: User Story 3 - See Where Runtime Is Getting Worse (Priority: P2)

Goal: Make slow-test drift visible through checked-in profiling and machine-readable reporting paths, plus initial documented runtime budgets.

Independent Test: Run the reporting and profiling commands and verify that they emit app-root-relative artifacts, show at least the top 10 slowest entries, and evaluate the run against documented lane budgets plus family thresholds.

Tests for User Story 3

T018 [P] [US3] Add at-least-top-10 JUnit, summary-artifact, family-budget, and report-shape coverage in apps/platform/tests/Unit/Support/TestLaneReportTest.php and apps/platform/tests/Feature/Guards/TestLaneArtifactsContractTest.php
T019 [P] [US3] Add profiling-mode, baseline-capture, and budget-status coverage in apps/platform/tests/Feature/Guards/ProfileLaneContractTest.php and apps/platform/tests/Unit/Support/TestLaneBudgetTest.php

Implementation for User Story 3

T020 [US3] Implement JUnit, summary, and budget artifact generation in scripts/platform-test-report, apps/platform/tests/Support/TestLaneReport.php, and apps/platform/storage/logs/test-lanes/.gitignore
T021 [US3] Add serial profiling and JUnit command paths in apps/platform/composer.json and scripts/platform-test-lane
T022 [US3] Capture the current full-suite baseline and the first measured lane baselines, including explicit verification that fast-feedback is at least 50% below the current full-suite baseline and confidence remains below the current full-suite baseline, in README.md and specs/206-test-suite-governance/quickstart.md
T023 [US3] Refine the heavy-governance selectors from profiling evidence and publish the initial lane budgets plus heavy-family thresholds in apps/platform/tests/Support/TestLaneManifest.php, README.md, and specs/206-test-suite-governance/quickstart.md

Checkpoint: The repository can emit machine-readable test artifacts, list at least the top 10 slowest entries, and compare runs against baseline-backed lane budgets plus family thresholds.

Phase 6: User Story 4 - Author Cheap, Honest Tests By Default (Priority: P2)

Goal: Stop the most common shared helper and one additional factory cluster from silently provisioning heavy tenant/provider graphs unless the test explicitly asks for them.

Independent Test: Use the shared helper and targeted factory states in isolation, verify that the default path stays minimal, and confirm that representative high-usage callers can opt into heavier behavior explicitly where needed.

Tests for User Story 4

T024 [P] [US4] Add shared-helper profile coverage for provider, credential, session, cache, workspace, membership, and UI-context opt-ins in apps/platform/tests/Unit/Support/CreateUserWithTenantProfilesTest.php and apps/platform/tests/Feature/Guards/FixtureCostProfilesGuardTest.php
T025 [P] [US4] Add minimal-versus-heavy factory state coverage in apps/platform/tests/Unit/Factories/TenantFactoryTest.php and apps/platform/tests/Unit/Factories/ProviderConnectionFactoryTest.php
T026 [P] [US4] Add taxonomy-audit and lane-placement guard coverage in apps/platform/tests/Feature/Guards/TestTaxonomyPlacementGuardTest.php and apps/platform/tests/Feature/Guards/ConfidenceLaneContractTest.php

Implementation for User Story 4

T027 [US4] Split createUserWithTenant() into minimal and explicit heavy, provider-enabled, credential-enabled, or UI-context profiles in apps/platform/tests/Pest.php
T028 [US4] Introduce explicit minimal and heavy state behavior for hidden workspace or provider graph creation in apps/platform/database/factories/TenantFactory.php and apps/platform/database/factories/ProviderConnectionFactory.php
T029 [US4] Audit and reclassify the first obvious heavy-governance batch in apps/platform/tests/Support/TestLaneManifest.php, covering apps/platform/tests/Architecture/PlatformVocabularyBoundaryGuardTest.php, apps/platform/tests/Architecture/ReasonTranslationPrimarySurfaceGuardTest.php, apps/platform/tests/Deprecation/IsPlatformSuperadminDeprecationTest.php, and apps/platform/tests/Feature/Guards/ActionSurfaceContractTest.php
T030 [US4] Migrate the first high-usage callers to the new fixture profiles in apps/platform/tests/Feature/Guards/ActionSurfaceContractTest.php, apps/platform/tests/Feature/OpsUx/OperateHubShellTest.php, apps/platform/tests/Feature/Findings/FindingExceptionRegisterTest.php, and apps/platform/tests/Feature/EntraAdminRoles/EntraAdminRolesFindingGeneratorTest.php
T031 [US4] Document honest taxonomy, DB reset rules, seed restrictions, and schema-baseline follow-up guidance in README.md and specs/206-test-suite-governance/quickstart.md

Checkpoint: Shared test setup defaults are materially cheaper, and the first obvious taxonomy misfits have been reclassified or regrouped with written guidance for future tests.

Phase 7: Polish & Cross-Cutting Concerns

Purpose: Reconcile the implementation with the design artifacts and validate the end-to-end workflow.

T032 [P] Reconcile the implemented lane model with specs/206-test-suite-governance/contracts/test-lane-manifest.schema.json, specs/206-test-suite-governance/contracts/test-suite-governance.logical.openapi.yaml, and specs/206-test-suite-governance/quickstart.md
T033 Run the focused validation flow documented in specs/206-test-suite-governance/quickstart.md
T034 Run formatting with cd apps/platform && ./vendor/bin/sail bin pint --dirty --format agent

Dependencies & Execution Order

Phase Dependencies

Setup (Phase 1): No dependencies, can start immediately.
Foundational (Phase 2): Depends on Setup completion and blocks all story work.
User Story 1 (Phase 3): Starts after Foundational completion.
User Story 2 (Phase 4): Starts after Foundational completion and can proceed in parallel with US1.
User Story 3 (Phase 5): Starts after Foundational completion; it benefits from US1 and US2 landing first so the lane inventory is stable.
User Story 4 (Phase 6): Starts after Foundational completion; it can proceed in parallel with US3 once the lane manifest and command seams exist.
Polish (Phase 7): Runs after the desired user stories are complete.

User Story Dependencies

User Story 1 (P1): No dependency on other stories. This is the recommended MVP slice.
User Story 2 (P1): Depends only on the foundational lane and command scaffolding, not on US1 completion.
User Story 3 (P2): Depends on the foundational lane and artifact scaffolding; it is strongest once US1 and US2 stabilize the lane inventory.
User Story 4 (P2): Depends on the foundational lane/guard scaffolding; it does not require US3 but shares docs and reporting outcomes with it.

Within Each User Story

Tests MUST be written and fail before implementation.
Lane manifest and command-routing seams must exist before story-specific lane work begins.
Artifact generation, baseline capture, and profiling refinement must exist before budget publication is considered complete.
Helper and factory profile changes must land before high-usage caller migrations are considered complete.
The first taxonomy audit must land before the initial reclassification batch is considered complete.

Parallel Opportunities

T002 and T003 can run in parallel.
T005 and T006 can run in parallel.
US1 test tasks T008 and T009 can run in parallel.
US2 test tasks T013 and T014 can run in parallel.
US3 test tasks T018 and T019 can run in parallel.
US4 test tasks T024, T025, and T026 can run in parallel.
After Foundational completion, US2, US3, and US4 can proceed in parallel with separate owners.

Parallel Example: User Story 1

# Launch the fast-feedback guard coverage together:
Task: "Add fast-feedback default-run contract coverage in apps/platform/tests/Feature/Guards/FastFeedbackLaneContractTest.php and apps/platform/tests/Feature/Guards/TestLaneCommandContractTest.php"
Task: "Add browser and heavy-family exclusion coverage for the default lane in apps/platform/tests/Feature/Guards/FastFeedbackLaneExclusionTest.php and apps/platform/tests/Feature/Guards/TestLaneManifestTest.php"

# Then land the fast lane implementation:
Task: "Define the fast-feedback lane selectors, parallel mode, and default-entry semantics in apps/platform/tests/Support/TestLaneManifest.php"
Task: "Wire the fast-feedback checked-in commands and default test alias behavior in apps/platform/composer.json and scripts/platform-test-lane"

Parallel Example: User Story 2

# Launch the broader-lane guard coverage together:
Task: "Add confidence-lane inclusion and heavy-lane separation coverage in apps/platform/tests/Feature/Guards/ConfidenceLaneContractTest.php and apps/platform/tests/Feature/Guards/HeavyGovernanceLaneContractTest.php"
Task: "Add browser-lane isolation and command-routing coverage in apps/platform/tests/Feature/Guards/BrowserLaneIsolationTest.php and apps/platform/tests/Feature/Guards/TestLaneCommandContractTest.php"

# Then land the lane separation work:
Task: "Define the confidence, heavy-governance, and browser lane selectors with baseline-backed budget metadata in apps/platform/tests/Support/TestLaneManifest.php"
Task: "Wire the checked-in confidence, browser, and heavy-governance commands in apps/platform/composer.json and scripts/platform-test-lane"

Parallel Example: User Story 3

# Launch the reporting coverage together:
Task: "Add at-least-top-10 JUnit, summary-artifact, family-budget, and report-shape coverage in apps/platform/tests/Unit/Support/TestLaneReportTest.php and apps/platform/tests/Feature/Guards/TestLaneArtifactsContractTest.php"
Task: "Add profiling-mode, baseline-capture, and budget-status coverage in apps/platform/tests/Feature/Guards/ProfileLaneContractTest.php and apps/platform/tests/Unit/Support/TestLaneBudgetTest.php"

# Then land the report generation and baseline work:
Task: "Implement JUnit, summary, and budget artifact generation in scripts/platform-test-report, apps/platform/tests/Support/TestLaneReport.php, and apps/platform/storage/logs/test-lanes/.gitignore"
Task: "Capture the current full-suite baseline and the first measured lane baselines in README.md and specs/206-test-suite-governance/quickstart.md"

Parallel Example: User Story 4

# Launch the cheap-default coverage together:
Task: "Add shared-helper profile coverage in apps/platform/tests/Unit/Support/CreateUserWithTenantProfilesTest.php and apps/platform/tests/Feature/Guards/FixtureCostProfilesGuardTest.php"
Task: "Add minimal-versus-heavy factory state coverage in apps/platform/tests/Unit/Factories/TenantFactoryTest.php and apps/platform/tests/Unit/Factories/ProviderConnectionFactoryTest.php"
Task: "Add taxonomy-audit and lane-placement guard coverage in apps/platform/tests/Feature/Guards/TestTaxonomyPlacementGuardTest.php and apps/platform/tests/Feature/Guards/ConfidenceLaneContractTest.php"

# Then land the fixture changes and taxonomy work:
Task: "Split createUserWithTenant() into minimal and explicit heavy or provider-enabled profiles in apps/platform/tests/Pest.php"
Task: "Audit and reclassify the first obvious heavy-governance batch in apps/platform/tests/Support/TestLaneManifest.php, covering apps/platform/tests/Architecture/PlatformVocabularyBoundaryGuardTest.php, apps/platform/tests/Architecture/ReasonTranslationPrimarySurfaceGuardTest.php, apps/platform/tests/Deprecation/IsPlatformSuperadminDeprecationTest.php, and apps/platform/tests/Feature/Guards/ActionSurfaceContractTest.php"

Implementation Strategy

MVP First (User Story 1 Only)

Complete Phase 1: Setup.
Complete Phase 2: Foundational.
Complete Phase 3: User Story 1.
Validate that the default contributor path now resolves to the fast-feedback lane and excludes browser plus heavy-governance families.

Incremental Delivery

Land the shared lane manifest, runner, and report scaffolding.
Ship the fast-feedback lane as the first visible contributor improvement.
Add the broader confidence lane plus separate browser and heavy-governance paths.
Add reporting, budgets, and profiling visibility.
Finish with cheap shared defaults and taxonomy guidance so new tests stop reintroducing hidden cost.

Parallel Team Strategy

One developer lands the foundational manifest and runner/report scaffolding.
A second developer can own fast-feedback and confidence lane separation while another owns reporting and budget output.
A final developer can drive the helper/factory cheap-default work and migrate the highest-usage caller files once the new fixture profiles exist.

Notes

[P] tasks are limited to work on different files with no incomplete dependency overlap.
US1 is the recommended MVP because it restores the default contributor loop first.
US2 turns the lane model into a full pre-merge workflow by separating confidence, browser, and heavy-governance paths.
US3 makes slow-test regressions visible and measurable.
US4 ensures the suite does not grow back into hidden heavy defaults after the lane model lands.

19 KiB Raw Blame History