ahmido
f03555eae1
## Summary - add the Spec 295 artifacts for full-suite failure classification and CI lane baseline work - fix `scripts/platform-test-artifacts` so Sail passes artifact staging inputs into the embedded PHP script via argv - add a guard test covering the artifact staging input contract ## Scope guards - no browser screenshot baselines included - no generated test artifacts included - no runtime application code changes included ## Notes - classification evidence and follow-up ownership are documented in `specs/295-full-suite-ci-baseline/failure-classification.md` - this PR is intentionally limited to the CI/lane/artifact contract slice for Spec 295 Co-authored-by: Ahmed Darrazi <ahmed.darrazi@live.de> Reviewed-on: #350
68 lines
4.5 KiB
Markdown
68 lines
4.5 KiB
Markdown
# Data Model: Full Suite Failure Classification & CI Lane Baseline
|
|
|
|
Spec `295` introduces no application entity, table, model, enum, migration, or persisted runtime artifact. Its only modeled data is spec-local workflow truth for implementation: a bounded failure-classification artifact and one final CI readiness decision.
|
|
|
|
## Spec-Local Artifact: `failure-classification.md`
|
|
|
|
`failure-classification.md` records the classification state for implementation. It is not product truth and must not be read by the application.
|
|
|
|
### Failure Group
|
|
|
|
| Field | Meaning |
|
|
|---|---|
|
|
| `Group` | One failing test file, assertion cluster, wrapper failure, report failure, artifact failure, budget breach, or environment failure sharing the same cause and owner |
|
|
| `Observed Command` | Exact command that produced the signal |
|
|
| `Seam` | One pinned CI/suite seam |
|
|
| `Category` | One pinned failure-classification category |
|
|
| `Observed Failure` | Concise description of the failure evidence |
|
|
| `Candidate Owner` | Existing script, support class, guard test, product area, or follow-up package likely responsible |
|
|
| `Fix In 295?` | `yes`, `no`, or `only-if-ci-contract-proven` |
|
|
| `Follow-up` | Follow-up spec/lane decision, or `none` when resolved |
|
|
| `Status` | `pending`, `classified`, `fixed`, `resolved`, `follow-up-required`, or `environment-blocked` |
|
|
|
|
## Pinned Categories
|
|
|
|
| Category | Meaning |
|
|
|---|---|
|
|
| `ci-signal-restored` | Full suite or lane split is green and usable as a CI signal |
|
|
| `ci-wrapper-or-manifest-regression` | Wrapper, composer script, workflow binding, or lane manifest no longer invokes the intended lane |
|
|
| `artifact-publication-regression` | Required report/JUnit/budget/profile/trend artifacts are not generated or staged as contracted |
|
|
| `budget-or-trend-baseline-drift` | Tests pass or mostly pass, but runtime budget/trend baseline output is stale or no longer interpretable |
|
|
| `product-runtime-or-test-regression` | A real app/test behavior failure outside the CI wrapper/report/artifact contract |
|
|
| `browser-lane-regression` | Existing browser lane or smoke failure that needs browser-specific follow-up unless it is a CI artifact issue |
|
|
| `flaky-or-environment` | Nondeterministic, local container, browser runtime, database, queue, or runner issue |
|
|
| `follow-up-spec-required` | Confirmed out-of-scope failure needing a separate spec/lane owner |
|
|
| `resolved-or-not-needed` | Initially suspected group that no longer needs work after rerun or adjacent classification |
|
|
|
|
## Pinned CI / Suite Seams
|
|
|
|
| Seam | Meaning |
|
|
|---|---|
|
|
| `raw-full-suite` | Direct `sail artisan test --compact` complete suite signal |
|
|
| `fast-feedback-lane` | Existing fast-feedback wrapper and manifest selection |
|
|
| `confidence-lane` | Existing confidence wrapper and manifest selection |
|
|
| `heavy-governance-lane` | Existing heavy-governance wrapper and manifest selection |
|
|
| `browser-lane` | Existing browser wrapper and smoke selection |
|
|
| `profiling-or-junit-support` | Support lanes used for profiling or durable machine-readable output |
|
|
| `lane-reporting` | `scripts/platform-test-report` and `TestLaneReport` output |
|
|
| `artifact-publication` | `scripts/platform-test-artifacts` and lane artifact contracts |
|
|
| `budget-trend-baseline` | `TestLaneBudget`, lane thresholds, and trend-history classification |
|
|
| `legacy-cutover-regression-guard` | Failures that appear to challenge the retired route/panel baseline from Specs `287` to `293` |
|
|
| `provider-verification-regression-guard` | Failures that appear to challenge Spec `294` provider/verification semantics |
|
|
|
|
## Final Readiness Decision
|
|
|
|
| Decision | Meaning |
|
|
|---|---|
|
|
| `restored-ci-signal` | Full suite and required lanes are green or have only non-blocking documented budget notes |
|
|
| `classified-follow-up-required` | CI is not fully green, but every remaining red group is classified with follow-up ownership |
|
|
| `blocked-by-environment` | Classification cannot complete because local/runner environment failures prevent trustworthy signal collection |
|
|
|
|
## Invariants
|
|
|
|
- Every failing group must use exactly one pinned category and one pinned seam.
|
|
- `ci-signal-restored` may be used only when the relevant command output is green.
|
|
- Product/runtime failures cannot be fixed in `295` unless the classification proves a direct CI/lane contract defect.
|
|
- Retired `/admin/t/...` or TenantPanelProvider behavior must never be restored as a remedy.
|
|
- The same category and seam names must appear in `spec.md`, `plan.md`, `tasks.md`, `quickstart.md`, `checklists/requirements.md`, and `failure-classification.md`.
|