TenantAtlas/specs/295-full-suite-ci-baseline/failure-classification.md

Failure Classification: Full Suite Failure Classification & CI Lane Baseline

Purpose

Use this artifact during implementation of Spec 295 to classify the complete platform suite signal after Specs 293 and 294.

This artifact is spec-local workflow truth only. It is not application runtime truth.

Implementation Scope Lock

• Date: 2026-05-11
• Branch: 295-full-suite-ci-baseline
• Baseline commit: eb85b76e Added Skill for Codex
• Pre-run working tree: only the active untracked spec directory specs/295-full-suite-ci-baseline/ is present; git diff --stat is empty.
• Scope confirmation: no runtime application code, Filament UI, routes, provider runtime, TenantPanelProvider behavior, /admin/t/... behavior, or completed Spec 293 / 294 artifacts are in scope unless a narrow CI/lane contract defect is proven by classification evidence.
• Forbidden repair confirmation: product/runtime failures, browser UI behavior failures, and provider/verification runtime failures are classification and follow-up candidates only unless the observed failure is directly caused by an existing lane wrapper, manifest, report, artifact, or budget/trend contract.

Pinned Failure-Classification Categories

Category	Meaning
ci-signal-restored	Full suite or lane split is green and usable as a CI signal
ci-wrapper-or-manifest-regression	Wrapper, composer script, workflow binding, or lane manifest no longer invokes the intended lane
artifact-publication-regression	Required report/JUnit/budget/profile/trend artifacts are not generated or staged as contracted
budget-or-trend-baseline-drift	Tests pass or mostly pass, but runtime budget/trend baseline output is stale or no longer interpretable
product-runtime-or-test-regression	A real app/test behavior failure outside the CI wrapper/report/artifact contract
browser-lane-regression	Existing browser lane or smoke failure needing browser-specific follow-up unless it is a CI artifact issue
flaky-or-environment	Nondeterministic, local container, browser runtime, database, queue, or runner issue
follow-up-spec-required	Confirmed out-of-scope failure needing a separate spec/lane owner
resolved-or-not-needed	Initially suspected group that no longer needs work after rerun or adjacent classification

Pinned CI / Suite Seams

Seam	Meaning
raw-full-suite	Direct sail artisan test --compact complete suite signal
fast-feedback-lane	Existing fast-feedback wrapper and manifest selection
confidence-lane	Existing confidence wrapper and manifest selection
heavy-governance-lane	Existing heavy-governance wrapper and manifest selection
browser-lane	Existing browser wrapper and smoke selection
profiling-or-junit-support	Support lanes used for profiling or durable machine-readable output
lane-reporting	scripts/platform-test-report and TestLaneReport output
artifact-publication	scripts/platform-test-artifacts and lane artifact contracts
budget-trend-baseline	TestLaneBudget, lane thresholds, and trend-history classification
legacy-cutover-regression-guard	Failures that appear to challenge the retired route/panel baseline from Specs 287 to 293
provider-verification-regression-guard	Failures that appear to challenge Spec 294 provider/verification semantics

Baseline Commands

Primary command:

export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && (cd apps/platform && ./vendor/bin/sail artisan test --compact)

Fallback lane split:

export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane fast-feedback
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane confidence
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane heavy-governance
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane browser

Report commands:

export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-report fast-feedback
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-report confidence
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-report heavy-governance
export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-report browser

Baseline Run Queue

The implementation must run or explicitly skip these targets and then add classified failure or success rows in the classification table below.

Run Target	Expected Command	Expected Seam	Current Status
raw-full-suite	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && (cd apps/platform && ./vendor/bin/sail artisan test --compact)	raw-full-suite	red; 450 failed, 8 skipped, 4194 passed, 28831 assertions, 4686.08s; output too broad/truncated for complete group ownership, fallback lane split required
fast-feedback-lane	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane fast-feedback	fast-feedback-lane	red; 82 failed, 1743 passed, 12151 assertions, 164.11s; report wall clock 171.551792s within 200s warning budget
confidence-lane	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane confidence	confidence-lane	red; 409 failed, 8 skipped, 3853 passed, 25994 assertions, 605.10s; report wall clock 622.531394s over 450s warning budget
heavy-governance-lane	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane heavy-governance	heavy-governance-lane	red; 21 failed, 319 passed, 2443 assertions, 314.28s; report wall clock 314.828382s within 315s warning budget
browser-lane	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane browser	browser-lane	red; 20 failed, 29 passed, 417 assertions, 285.32s; report wall clock 285.719479s over 150s warning budget

Classification Table

The implementation must append one row per failing group or one ci-signal-restored row for a fully green signal.

Group	Observed Command	Seam	Category	Observed Failure	Candidate Owner	Fix In 295?	Follow-up	Status
raw-full-suite-red-baseline	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && (cd apps/platform && ./vendor/bin/sail artisan test --compact)	raw-full-suite	follow-up-spec-required	Raw full suite completed but is not a restored CI signal: 450 failed, 8 skipped, 4194 passed, 28831 assertions, 4686.08s. Failure output includes unit RBAC/capability assertions, provider boundary/start gate failures, route generation errors for workspace-aware operation routes, Filament panel URL generation errors, and browser smoke failures; full output was too broad/truncated to classify every group from raw output alone.	suite/lane ownership classification	no	run fallback lane split and classify by lane/report artifacts before any repair	classified
fast-feedback-lane-product-red	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane fast-feedback	fast-feedback-lane	product-runtime-or-test-regression	Lane completed through the wrapper but is red: 82 failed, 1743 passed, 12151 assertions, 164.11s. JUnit and console output show workspace-aware operation route URL generation without workspace, Filament hasTenancy() calls with no panel context, authorization expectation drift, RBAC/UI action assertions, provider boundary/start-gate assertions, and monitoring/required-permissions surfaces.	workspace route and Filament panel-context follow-up; RBAC/authorization follow-up; provider verification follow-up	no	split product/test failures into focused follow-up specs; keep 295 limited to lane/report/artifact contracts	classified
confidence-lane-product-red	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane confidence	confidence-lane	product-runtime-or-test-regression	Lane completed through the wrapper but is red: 409 failed, 8 skipped, 3853 passed, 25994 assertions, 605.10s. Failure groups include the same workspace-route and Filament panel-context errors, missing/renamed Filament resource routes, bulk-action test helpers returning null actions, deny-as-not-found expectation drift, and legacy admin URL assumptions.	confidence-lane product/runtime owners by resource area	no	create follow-up ownership slices before any product repair; do not absorb broad application repair into 295	classified
heavy-governance-lane-product-red	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane heavy-governance	heavy-governance-lane	product-runtime-or-test-regression	Lane completed through the wrapper but is red: 21 failed, 319 passed, 2443 assertions, 314.28s. Failures are concentrated in canonical operation detail/list tests missing the workspace route parameter, Filament URL generation without panel context, one tenant sync summary-count assertion, and RBAC relation-manager UI enforcement.	operations canonical viewer/list follow-up; tenant sync summary follow-up; RBAC UI follow-up	no	follow-up specs should repair these product/test contracts independently of the 295 lane baseline	classified
browser-lane-red	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane browser	browser-lane	browser-lane-regression	Browser lane completed through the wrapper but is red: 20 failed, 29 passed, 417 assertions, 285.32s. Failures include smoke-login pages not showing Dashboard, workspace-aware operation route URL generation errors, Filament hasTenancy() panel-context errors, a tenant dashboard layout assertion, a Spec 279 /admin/t/... path expectation, and a tenant membership page copy/action expectation.	browser smoke/product UI follow-up owners	no	split browser repairs separately; do not treat the lane as green or restore retired tenant routes in 295	classified
legacy-cutover-route-expectations	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane browser and export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane confidence	legacy-cutover-regression-guard	follow-up-spec-required	Browser Spec 279 still expects /admin/t/spec-279-production, while the current path is /admin/workspaces/{workspace}/environments/{environment}. Confidence output also includes older expectations around /admin/operations and admin operation URLs.	tenant cutover regression guard owner	no	create follow-up only if current cutover truth should change; do not restore /admin/t/..., TenantPanelProvider behavior, or historical compatibility routes in 295	classified
provider-verification-regression-guard	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-lane fast-feedback and raw full suite	provider-verification-regression-guard	follow-up-spec-required	Raw and fast-feedback output include provider boundary/status assertions such as unexpected provider.capability_registry, review_required vs blocked, and provider operation start-gate dispatch count drift. These are provider/runtime semantics, not lane wrapper failures.	Spec 294 provider/verification follow-up owner	no	open a provider verification follow-up if the current runtime semantics are wrong; do not rewrite Spec 294 artifacts under 295	classified
lane-reporting-all-lanes	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-report fast-feedback, confidence, heavy-governance, browser	lane-reporting	resolved-or-not-needed	All four report commands exited 0 and rendered summary.md, report.json, budget.json, junit.xml, and trend-history.json references. Reports preserved the red lane status and exposed budget/trend metadata instead of crashing.	TestLaneReport and report wrapper	no	none for report rendering	classified
budget-trend-baseline-status	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-report fast-feedback, confidence, heavy-governance, browser	budget-trend-baseline	resolved-or-not-needed	Fast-feedback reported within-budget at 171.551792s under 200s. Heavy-governance reported within-budget at 314.828382s under 315s. Confidence and browser reported warning-level budget output, respectively 622.531394s over 450s and 285.719479s over 150s, with warning enforcement and no hard budget-blocking failure. Trend windows were either stable, scope-changed, or insufficient-history as documented.	TestLaneBudget/trend baseline owner	no	investigate confidence/browser runtime separately if desired; no 295 budget relaxation or baseline rewrite was justified	classified
artifact-publication-env-forwarding	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-artifacts fast-feedback /tmp/tenantpilot-295-fast-feedback-artifacts and matching commands for confidence, heavy-governance, browser	artifact-publication	artifact-publication-regression	Initial artifact staging failed for every lane with Unknown test lane [] because scripts/platform-test-artifacts passed lane and staging inputs to the Sail PHP process via host environment variables that were empty inside the container.	scripts/platform-test-artifacts	yes	fixed by passing lane, staging directory, and artifact directory as PHP argv through Sail and adding a guard test	resolved
artifact-publication-after-fix	export PATH="/bin:/usr/bin:/usr/local/bin:$PATH" && ./scripts/platform-test-artifacts fast-feedback /tmp/tenantpilot-295-fast-feedback-artifacts and matching commands for confidence, heavy-governance, browser	artifact-publication	resolved-or-not-needed	After the wrapper fix, all four artifact commands exited 0 and staged all five required artifacts with complete: true, primaryFailureClassId: null, and no missing required artifacts.	scripts/platform-test-artifacts and TestLaneReport artifact contract	yes	none for artifact publication	classified
junit-support-output	not run separately; lane wrappers produced apps/platform/storage/logs/test-lanes/*-latest.junit.xml	profiling-or-junit-support	resolved-or-not-needed	Separate ./scripts/platform-test-lane junit was not needed because fast-feedback, confidence, heavy-governance, and browser wrappers already produced machine-readable JUnit artifacts used for classification.	TestLaneManifest JUnit support	no	none unless a future follow-up needs the dedicated JUnit lane	classified

Final Readiness Decision

Current decision: classified-follow-up-required

Allowed values:

• restored-ci-signal
• classified-follow-up-required
• blocked-by-environment

Classification Rules

• Every red group must have exactly one pinned category and one pinned seam.
• Do not use ci-signal-restored for a partially red lane.
• Fix in 295 only if the group is directly tied to CI wrapper, manifest, report, artifact, or budget/trend contract drift.
• Split product/runtime failures to follow-up ownership.
• Do not restore TenantPanelProvider, /admin/t/..., or retired tenant-scoped fallback routes.
• Do not rewrite completed Specs 293 or 294.