ahmido
f03555eae1
## Summary - add the Spec 295 artifacts for full-suite failure classification and CI lane baseline work - fix `scripts/platform-test-artifacts` so Sail passes artifact staging inputs into the embedded PHP script via argv - add a guard test covering the artifact staging input contract ## Scope guards - no browser screenshot baselines included - no generated test artifacts included - no runtime application code changes included ## Notes - classification evidence and follow-up ownership are documented in `specs/295-full-suite-ci-baseline/failure-classification.md` - this PR is intentionally limited to the CI/lane/artifact contract slice for Spec 295 Co-authored-by: Ahmed Darrazi <ahmed.darrazi@live.de> Reviewed-on: #350
91 lines
4.0 KiB
Markdown
91 lines
4.0 KiB
Markdown
# Quickstart: Full Suite Failure Classification & CI Lane Baseline
|
|
|
|
## Purpose
|
|
|
|
Use this package to classify whether the complete platform test suite is a reliable CI signal after Specs `293` and `294`.
|
|
|
|
## Before Implementation
|
|
|
|
1. Review:
|
|
- `/Users/ahmeddarrazi/Documents/projects/wt-plattform/specs/295-full-suite-ci-baseline/spec.md`
|
|
- `/Users/ahmeddarrazi/Documents/projects/wt-plattform/specs/295-full-suite-ci-baseline/plan.md`
|
|
- `/Users/ahmeddarrazi/Documents/projects/wt-plattform/specs/295-full-suite-ci-baseline/data-model.md`
|
|
- `/Users/ahmeddarrazi/Documents/projects/wt-plattform/specs/295-full-suite-ci-baseline/failure-classification.md`
|
|
2. Confirm the branch is clean.
|
|
3. Confirm no implementation step is about restoring TenantPanelProvider, `/admin/t/...`, or tenant-scoped legacy fallbacks.
|
|
|
|
## Primary Classification Flow
|
|
|
|
Use only the pinned categories and seams from `failure-classification.md`: `ci-signal-restored`, `ci-wrapper-or-manifest-regression`, `artifact-publication-regression`, `budget-or-trend-baseline-drift`, `product-runtime-or-test-regression`, `browser-lane-regression`, `flaky-or-environment`, `follow-up-spec-required`, `resolved-or-not-needed`; and `raw-full-suite`, `fast-feedback-lane`, `confidence-lane`, `heavy-governance-lane`, `browser-lane`, `profiling-or-junit-support`, `lane-reporting`, `artifact-publication`, `budget-trend-baseline`, `legacy-cutover-regression-guard`, `provider-verification-regression-guard`.
|
|
|
|
Run the raw full suite when feasible:
|
|
|
|
```bash
|
|
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && (cd apps/platform && ./vendor/bin/sail artisan test --compact)
|
|
```
|
|
|
|
Record the outcome in `failure-classification.md`.
|
|
|
|
If the raw full suite is too slow, noisy, or environment-blocked to classify, run the explicit lane split:
|
|
|
|
```bash
|
|
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane fast-feedback
|
|
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane confidence
|
|
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane heavy-governance
|
|
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane browser
|
|
```
|
|
|
|
## CI Report and Artifact Flow
|
|
|
|
After lane runs, generate lane reports when needed:
|
|
|
|
```bash
|
|
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-report fast-feedback
|
|
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-report confidence
|
|
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-report heavy-governance
|
|
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-report browser
|
|
```
|
|
|
|
Use artifact staging only if artifact publication itself is being validated:
|
|
|
|
```bash
|
|
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-artifacts fast-feedback /tmp/tenantpilot-fast-feedback-artifacts
|
|
```
|
|
|
|
## Fix Rules
|
|
|
|
Fix in `295` only when the failure is directly and narrowly caused by:
|
|
|
|
- `scripts/platform-test-lane`
|
|
- `scripts/platform-test-report`
|
|
- `scripts/platform-test-artifacts`
|
|
- `apps/platform/tests/Support/TestLaneManifest.php`
|
|
- `apps/platform/tests/Support/TestLaneReport.php`
|
|
- `apps/platform/tests/Support/TestLaneBudget.php`
|
|
- directly related CI guard tests under `apps/platform/tests/Feature/Guards/`
|
|
|
|
Do not fix in `295` when the failure requires:
|
|
|
|
- application runtime behavior changes
|
|
- Filament page/resource changes
|
|
- routes, middleware, policies, services, jobs, migrations, views, or models
|
|
- provider/verification runtime changes beyond the completed Spec `294`
|
|
- browser UI repair
|
|
- tenant-cutover compatibility restoration
|
|
|
|
Classify those as follow-up work instead.
|
|
|
|
## Expected Close-Out
|
|
|
|
Close out with exactly one final readiness decision:
|
|
|
|
- `restored-ci-signal`
|
|
- `classified-follow-up-required`
|
|
- `blocked-by-environment`
|
|
|
|
Then run formatting for any changed PHP files:
|
|
|
|
```bash
|
|
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && (cd apps/platform && ./vendor/bin/sail bin pint --dirty --format agent)
|
|
```
|