TenantAtlas/specs/210-ci-matrix-budget-enforcement/quickstart.md
ahmido bf38ec1780
Some checks failed
Main Confidence / confidence (push) Failing after 3m36s
Spec 210: implement CI test matrix budget enforcement (#243)
## Summary
- add explicit Gitea workflow files for PR Fast Feedback, `dev` Confidence, Heavy Governance, and Browser lanes
- extend the repo-truth lane support seams with workflow profiles, trigger-aware budget enforcement, artifact publication contracts, CI summaries, and failure classification
- add deterministic artifact staging, new CI governance guard coverage, and Spec 210 planning/contracts/docs updates

## Validation
- `cd apps/platform && ./vendor/bin/sail bin pint --dirty --format agent`
- `cd apps/platform && ./vendor/bin/sail artisan test --compact tests/Feature/Guards/CiFastFeedbackWorkflowContractTest.php tests/Feature/Guards/CiConfidenceWorkflowContractTest.php tests/Feature/Guards/CiHeavyBrowserWorkflowContractTest.php tests/Feature/Guards/CiLaneFailureClassificationContractTest.php tests/Feature/Guards/FastFeedbackLaneContractTest.php tests/Feature/Guards/ConfidenceLaneContractTest.php tests/Feature/Guards/HeavyGovernanceLaneContractTest.php tests/Feature/Guards/BrowserLaneIsolationTest.php tests/Feature/Guards/FixtureLaneImpactBudgetTest.php tests/Feature/Guards/TestLaneManifestTest.php tests/Feature/Guards/TestLaneArtifactsContractTest.php tests/Feature/Guards/TestLaneCommandContractTest.php`
- `./scripts/platform-test-lane fast-feedback`
- `./scripts/platform-test-lane confidence`
- `./scripts/platform-test-lane heavy-governance`
- `./scripts/platform-test-lane browser`
- `./scripts/platform-test-report fast-feedback`
- `./scripts/platform-test-report confidence`

## Notes
- scheduled Heavy Governance and Browser workflows stay gated behind `TENANTATLAS_ENABLE_HEAVY_GOVERNANCE_SCHEDULE=1` and `TENANTATLAS_ENABLE_BROWSER_SCHEDULE=1`
- the remaining rollout evidence task is capturing the live Gitea run set this PR enables: PR Fast Feedback, `dev` Confidence, manual and scheduled Heavy Governance, and manual and scheduled Browser runs

Co-authored-by: Ahmed Darrazi <ahmed.darrazi@live.de>
Reviewed-on: #243
2026-04-17 18:04:35 +00:00

7.1 KiB

Quickstart: CI Test Matrix & Runtime Budget Enforcement

Preconditions

  • Repository Actions are enabled in Gitea for this repository.
  • At least one trusted act_runner is registered with a single stable runner label.
  • The runner can reach the Gitea instance and the action source used by the workflows.
  • dev remains the integration branch for broader shared validation.
  • The existing local lane wrappers still pass before CI wiring begins.

Required Validation Evidence

  • One representative pull_request Fast Feedback run.
  • One representative push to dev Confidence run.
  • One manual Heavy Governance workflow run.
  • One scheduled Heavy Governance workflow run after schedules are enabled.
  • One manual Browser workflow run.
  • One scheduled Browser workflow run after schedules are enabled.
  • Each evidence record must capture the trigger, executed lane, artifact bundle names, budget outcome class, and primary failure class or explicit clean-success result. The Fast Feedback evidence must reference the chosen CI variance tolerance, and any material runtime recalibration must be recorded in specs/210-ci-matrix-budget-enforcement/spec.md or the active PR and may be linked from the affected evidence records.

Frozen Trigger Matrix

Trigger Workflow Lane Expected artifact bundle
pull_request (opened, reopened, synchronize) test-pr-fast-feedback.yml fast-feedback summary, report, budget, JUnit
push to dev test-main-confidence.yml confidence summary, report, budget, JUnit
workflow_dispatch heavy test-heavy-governance.yml heavy-governance summary, report, budget, JUnit
scheduled heavy test-heavy-governance.yml heavy-governance summary, report, budget, JUnit
workflow_dispatch browser test-browser.yml browser summary, report, budget, JUnit
scheduled browser test-browser.yml browser summary, report, budget, JUnit

No-New-Fixture-Cost Rule

  • Keep CI-governance additions limited to manifest or budget metadata, lightweight guard tests, workflow files, and the artifact staging helper.
  • Do not add shared product fixtures, broaden default guard setup, or move CI-governance checks into Heavy Governance or Browser by default.
  • Treat the Fast Feedback CI variance allowance as 15s; a pull request lane run between 200s and 215s is warning-only, while a run above 215s becomes a blocking budget breach.

Artifact Locations

  • Lane-local machine-readable outputs remain under apps/platform/storage/logs/test-lanes.
  • CI upload staging directories are .gitea-artifacts/pr-fast-feedback, .gitea-artifacts/main-confidence, .gitea-artifacts/heavy-governance, and .gitea-artifacts/browser.
  • Scheduled Heavy Governance is gated by TENANTATLAS_ENABLE_HEAVY_GOVERNANCE_SCHEDULE=1 and scheduled Browser is gated by TENANTATLAS_ENABLE_BROWSER_SCHEDULE=1.

Implementation Order

  1. Extend repo truth first.

    • Add trigger-aware CI policy fields to TestLaneManifest.
    • Extend TestLaneBudget and TestLaneReport with failure-class and CI-summary metadata.
    • Add scripts/platform-test-artifacts for deterministic export and upload staging.
  2. Land the blocking pull request path.

    • Add .gitea/workflows/test-pr-fast-feedback.yml.
    • Use only scripts/platform-test-lane fast-feedback.
    • Stage and upload the Fast Feedback artifacts.
  3. Land the broader mainline path.

    • Add .gitea/workflows/test-main-confidence.yml.
    • Trigger it on push to dev.
    • Publish the Confidence lane JUnit XML, summary, report, and budget outputs from the same run.
  4. Land the expensive lanes as separate workflows.

    • Add .gitea/workflows/test-heavy-governance.yml for workflow_dispatch, with the schedule enabled only after the first successful manual validation.
    • Add .gitea/workflows/test-browser.yml for workflow_dispatch, with the schedule enabled only after the first successful manual validation.
    • Keep their failure semantics non-blocking at first unless stability evidence justifies more.
  5. Add guards and documentation.

    • Add CI-governance guard tests, including wrong-lane and unresolved-entry-point failure classification.
    • Update README.md with trigger policy, local reproduction, artifact locations, and blocking semantics.

Local Validation Commands

Run the existing wrappers before pushing workflow changes:

./scripts/platform-test-lane fast-feedback
./scripts/platform-test-lane confidence
./scripts/platform-test-lane heavy-governance
./scripts/platform-test-lane browser
./scripts/platform-test-report fast-feedback
./scripts/platform-test-report confidence

Run the focused guard suites after implementation changes:

cd apps/platform && ./vendor/bin/sail artisan test --compact tests/Feature/Guards/CiFastFeedbackWorkflowContractTest.php tests/Feature/Guards/CiConfidenceWorkflowContractTest.php tests/Feature/Guards/CiHeavyBrowserWorkflowContractTest.php tests/Feature/Guards/CiLaneFailureClassificationContractTest.php tests/Feature/Guards/FastFeedbackLaneContractTest.php tests/Feature/Guards/ConfidenceLaneContractTest.php tests/Feature/Guards/HeavyGovernanceLaneContractTest.php tests/Feature/Guards/BrowserLaneIsolationTest.php tests/Feature/Guards/FixtureLaneImpactBudgetTest.php tests/Feature/Guards/TestLaneManifestTest.php tests/Feature/Guards/TestLaneArtifactsContractTest.php tests/Feature/Guards/TestLaneCommandContractTest.php

First Live Gitea Validation

  1. Push the workflow files and helper changes on a feature branch.
  2. Open or update a pull request and confirm only the Fast Feedback workflow triggers.
  3. Verify the uploaded artifact bundle contains lane-specific summary, report, budget, and JUnit files.
  4. Push or merge to dev and confirm the Confidence workflow triggers and publishes the broader artifact set.
  5. Manually dispatch Heavy Governance and Browser workflows and confirm they remain separate.
  6. Review the manual run summaries and guard results to confirm failure classification stays legible and the CI-governance changes did not widen fast-path fixture cost or lane membership unexpectedly.
  7. Enable schedules only after the manual runs succeed and their artifact bundles are legible.
  8. Capture one scheduled Heavy Governance run and one scheduled Browser run, then record trigger, executed lane, bundle names, budget outcome, and the primary failure class or explicit clean-success result in this quickstart, and record the Fast Feedback variance tolerance or any material recalibration note in specs/210-ci-matrix-budget-enforcement/spec.md or the active PR.

Review Checklist

  • Pull request workflow uses the repo wrapper and no inline test paths.
  • dev push workflow publishes Confidence artifacts from one lane run.
  • Heavy Governance and Browser have separate workflow files and separate upload bundles.
  • Artifact names are deterministic per run and per lane.
  • Failure summaries distinguish test, budget, artifact, wrapper, manifest, and infrastructure problems.
  • README explains what blocks a pull request and what only warns.
  • CI-governance changes do not widen default fixtures or promote guard coverage into Heavy Governance or Browser unexpectedly.