ahmido
38523814c2
## Summary - restore broad full-suite green-signal coverage across platform governance, operations, onboarding, dashboard/productization, and customer review flows - align related platform tests and supporting behavior with the current expected state for this restoration pass - update the spec-candidates queue as part of the same suite-restoration sweep ## Validation - `cd apps/platform && ./vendor/bin/sail artisan test --compact tests/Browser/Dashboard/TenantDashboardProductizationSmokeTest.php tests/Browser/Reviews/CustomerReviewWorkspaceSmokeTest.php tests/Browser/Spec194GovernanceFrictionSmokeTest.php tests/Browser/Spec265DecisionRegisterSmokeTest.php` Co-authored-by: Ahmed Darrazi <ahmed.darrazi@live.de> Reviewed-on: #351
130 lines
24 KiB
Markdown
130 lines
24 KiB
Markdown
# Failure Inventory: Full Suite Green Signal Restoration
|
|
|
|
## Purpose
|
|
|
|
Track every observed red group during Spec 296 implementation. This artifact is spec-local workflow evidence only; it is not application runtime truth.
|
|
|
|
## Baseline Required Commands
|
|
|
|
Primary:
|
|
|
|
```bash
|
|
cd apps/platform
|
|
./vendor/bin/sail artisan test --compact
|
|
```
|
|
|
|
Fallback lanes when raw output is too broad or truncated:
|
|
|
|
```bash
|
|
./scripts/platform-test-lane fast-feedback
|
|
./scripts/platform-test-lane confidence
|
|
./scripts/platform-test-lane heavy-governance
|
|
./scripts/platform-test-lane browser
|
|
```
|
|
|
|
## Pinned Classifications
|
|
|
|
- `stale-test-expectation`
|
|
- `missing-fixture`
|
|
- `route-context-drift`
|
|
- `panel-context-drift`
|
|
- `rbac-contract-drift`
|
|
- `provider-boundary-drift`
|
|
- `browser-lane-drift`
|
|
- `true-runtime-bug`
|
|
- `obsolete-test`
|
|
- `wrong-lane`
|
|
- `environment/flaky`
|
|
- `fixed`
|
|
|
|
## Fix Types
|
|
|
|
- `test-rebaseline`
|
|
- `fixture-repair`
|
|
- `route-parameter-repair`
|
|
- `panel-context-repair`
|
|
- `rbac-assertion-repair`
|
|
- `provider-fixture-or-contract-repair`
|
|
- `browser-expectation-repair`
|
|
- `small-runtime-fix`
|
|
- `lane-move`
|
|
- `skip-with-reason`
|
|
- `obsolete-removal`
|
|
- `no-fix-needed`
|
|
|
|
## Inventory Table
|
|
|
|
## Current Spec 296 Safety Gate
|
|
|
|
- Branch: `296-full-suite-green-signal-restoration`
|
|
- Initial `git status --short`: only untracked active spec directory `?? specs/296-full-suite-green-signal-restoration/`
|
|
- Initial `git diff --stat`: empty
|
|
- Baseline commit noted before implementation: `eca92364 Merge remote-tracking branch 'origin/platform-dev' into platform-dev`
|
|
- Scope decision: the untracked files are spec-local preparation artifacts for the active branch; no unrelated uncommitted application/runtime changes were present.
|
|
|
|
## Final Spec 296 Validation Status
|
|
|
|
- Raw full-suite final rerun: not executed after repairs. The first raw baseline remains recorded below as the original red signal (`450 failed, 8 skipped, 4195 passed`) and was too broad/long-running to use as the final bounded loop proof in this continuation.
|
|
- Fast-feedback lane: green, `1828 passed`, `12517 assertions`, `230.90s` command duration, `240.33s` lane wall clock, CI outcome `passed / informational`.
|
|
- Confidence lane: green, `4265 passed`, `8 skipped`, `28030 assertions`, `1591.67s` command duration, `1613.02s` lane wall clock, CI outcome `passed / informational`.
|
|
- Heavy-governance lane: green, `340 passed`, `2525 assertions`, `321.82s` command duration, `322.13s` lane wall clock, CI outcome `passed / informational`.
|
|
- Browser lane: green, `49 passed`, `837 assertions`, `258.63s` command duration, `259.05s` lane wall clock, CI outcome `passed / informational`.
|
|
- Spec 288 guard lane: green, `50 passed`, `2055 assertions`, `18.23s`.
|
|
- Spec 293 cutover lane: green, `127 passed`, `908 assertions`, `61.82s`.
|
|
- Spec 294 ProviderConnections/Verification lane: green, `109 passed`, `782 assertions`, `56.74s`.
|
|
- Focused confidence-failure regression set: green, `262 passed`, `1530 assertions`, `222.07s`.
|
|
- Focused remaining cluster set: green, `86 passed`, `490 assertions`, `52.27s`.
|
|
- Focused four-failure rerun: green, `4 passed`, `22 assertions`, `4.39s`.
|
|
- Final classification: no current in-scope lane red group remains. The only unresolved proof item is the optional long raw-suite rerun, which is documented as not executed rather than claimed green.
|
|
|
|
Historic baseline rows below are preserved as first-observed failure evidence. Their "current baseline" wording reflects the time each row was recorded, not the final Spec 296 lane state.
|
|
|
|
| Test file | Test name | Failure summary | First observed command | Owner area | Classification | Fix type | Fixed now? yes/no | Follow-up required? yes/no | Validation command | Final status |
|
|
|---|---|---|---|---|---|---|---|---|---|---|
|
|
| Final lane split | fast-feedback | Final lane split passed after route, panel context, RBAC, provider fixture, and browser/test expectation repairs. | `./scripts/platform-test-lane fast-feedback` | fast-feedback lane | `fixed` | `no-fix-needed` | yes | no | `./scripts/platform-test-lane fast-feedback` | green: 1828 passed, 12517 assertions |
|
|
| Final lane split | confidence | Final confidence lane passed after focused cluster repairs and regression reruns. | `./scripts/platform-test-lane confidence` | confidence lane | `fixed` | `no-fix-needed` | yes | no | `./scripts/platform-test-lane confidence` | green: 4265 passed, 8 skipped, 28030 assertions |
|
|
| Final lane split | heavy-governance | Final heavy-governance lane passed and remained within the current 325s budget. | `./scripts/platform-test-lane heavy-governance` | heavy-governance lane | `fixed` | `no-fix-needed` | yes | no | `./scripts/platform-test-lane heavy-governance` | green: 340 passed, 2525 assertions |
|
|
| Final lane split | browser | Final browser lane passed. Screenshot deletions generated by the run were restored and are not committed. | `./scripts/platform-test-lane browser` | browser lane | `fixed` | `no-fix-needed` | yes | no | `./scripts/platform-test-lane browser`; `git status --short -- apps/platform/tests/Browser/Screenshots` | green: 49 passed, 837 assertions; screenshot directory clean |
|
|
| Final guard lane | Spec 288 | Cutover/provider/browser-lane/no-role-string guard stayed green. | Spec 288 guard command from `spec.md` | guard lane | `fixed` | `no-fix-needed` | yes | no | Spec 288 guard command from `spec.md` | green: 50 passed, 2055 assertions |
|
|
| Final guard lane | Spec 293 | Cutover regression lane stayed green without restoring TenantPanelProvider or retired route compatibility. | Spec 293 command from `spec.md` | guard lane | `fixed` | `no-fix-needed` | yes | no | Spec 293 command from `spec.md` | green: 127 passed, 908 assertions |
|
|
| Final guard lane | ProviderConnections/Verification | Provider and verification semantics stayed green. | `./vendor/bin/sail artisan test --compact tests/Feature/ProviderConnections tests/Feature/Verification` | guard lane | `fixed` | `no-fix-needed` | yes | no | `./vendor/bin/sail artisan test --compact tests/Feature/ProviderConnections tests/Feature/Verification` | green: 109 passed, 782 assertions |
|
|
| Final raw suite | raw full-suite rerun | Not rerun after repairs. The active proof set is the lane split plus guard commands above; do not claim raw-suite green from this artifact. | N/A after repairs | suite governance | `fixed` | `no-fix-needed` for lane split; raw rerun remains optional long proof | no | yes | `cd apps/platform && ./vendor/bin/sail artisan test --compact` | not run final; lane split green |
|
|
| Raw full suite | Spec 296 current baseline | Current raw suite is red: 450 failed, 8 skipped, 4195 passed, 28838 assertions, 4727.04s. Tool output was too broad/truncated (`10285` output lines), so lane splits are required before repair. Visible groups include capability/RBAC unit drift, provider-boundary drift, workspace operation route parameters, Filament panel context errors, browser smoke login failures, and stale `/admin/t/...` expectation. | `cd apps/platform && ./vendor/bin/sail artisan test --compact` | suite governance | `route-context-drift` | `no-fix-needed` until lane split classification | no | yes | `./scripts/platform-test-lane fast-feedback`; `./scripts/platform-test-lane confidence`; `./scripts/platform-test-lane heavy-governance`; `./scripts/platform-test-lane browser` | current baseline recorded; fallback lane splits required |
|
|
| Lane split | fast-feedback current baseline | Current fast-feedback lane is red: 82 failed, 1744 passed, 12158 assertions, 220.89s. Visible groups include missing `workspace` parameter for `admin.operations.view`, Filament `hasTenancy()` with null panel from `WorkspaceScopedTenantRoutes`, `/admin/operations` 404 against stale route expectations, Livewire dashboard drillthrough query-state drift, onboarding/session deny-as-not-found drift, provider boundary status drift, tenant page category drift, and UI enforcement bulk authorization drift. Output was broad enough that JUnit/report artifacts should be used for exact per-file follow-up before repair. | `./scripts/platform-test-lane fast-feedback` | fast-feedback lane | `route-context-drift` | `no-fix-needed` until all lane baselines complete | no | yes | Focused reproductions after confidence/heavy/browser baselines are recorded | current lane baseline recorded |
|
|
| Lane split | confidence current baseline | Current confidence lane is red: 409 failed, 8 skipped, 3854 passed, 26001 assertions, command duration 659.75s; lane artifact wall clock 677.29s and reports `failed / blocking` with budget warning. Visible groups include stale/non-member 404 vs 200 authorization expectations, many Filament `hasTenancy()` null-panel errors, missing `workspace` parameters for `admin.operations.*`, retired backup-schedule/resource route names, table bulk-action helper null instances, stale `filament.tenant.*` resource route expectations, workspace overview copy/query-state drift, provider operation result drift, and dashboard/detail continuity assertion drift. | `./scripts/platform-test-lane confidence` | confidence lane | `panel-context-drift` | `no-fix-needed` until all lane baselines complete | no | yes | Focused reproductions after heavy/browser baselines are recorded | current lane baseline recorded |
|
|
| Lane split | heavy-governance current baseline | Current heavy-governance lane is red: 21 failed, 319 passed, 2443 assertions, command duration 319.88s; lane artifact wall clock 320.35s and reports `failed / blocking` within the 325s budget. Failure groups are concentrated in workspace-aware operation route generation (`admin.operations.index` and `admin.operations.view` missing `workspace`) across Spec 078/144 operation detail/list tests, one canonical deep-link mismatch copy assertion, one Filament `hasTenancy()` null-panel error in `ActivityFeedbackSurfaceTest`, one tenant sync summary-count assertion (`succeeded` null vs 1), and one BackupItems relation-manager UI action visibility assertion. | `./scripts/platform-test-lane heavy-governance` | heavy-governance lane | `route-context-drift` | `no-fix-needed` until browser baseline and grouped classification complete | no | yes | Focused operations, panel-context, summary-count, and relation-manager tests after browser baseline is recorded | current lane baseline recorded |
|
|
| Lane split | browser current baseline | Current browser lane is red: 20 failed, 29 passed, 417 assertions, command duration 293.75s; lane artifact wall clock 294.26s and reports `failed / blocking` with budget warning. Failure groups include smoke-login pages not showing `Dashboard`, workspace-aware `admin.operations.index/view` route generation missing `workspace`, Filament `hasTenancy()` null-panel URL generation, stale `/admin/t/spec-279-production` path expectation after managed-environment cutover, tenant dashboard visual spacing assertion drift, and tenant memberships copy/action drift. Ten tracked browser screenshots are dirty and copied to `/tmp/tenantpilot-296-browser-evidence/browser-lane-current/` as evidence-only. | `./scripts/platform-test-lane browser` | browser lane | `browser-lane-drift` | `no-fix-needed` until grouped classification complete | no | yes | Browser focused files after route/panel/copy repairs; `git status --short apps/platform/tests/Browser/Screenshots` before screenshot cleanup | current lane baseline recorded |
|
|
| Root cause group | Workspace operations route context | Duplicate failures across fast-feedback, confidence, heavy-governance, and browser generate `admin.operations.index` or `admin.operations.view` without the required `workspace` route parameter, or still request retired `/admin/operations` URLs as current 200 surfaces. Representative files include `OperatorExplanationSurfaceAuthorizationTest`, `ReasonTranslationScopeSafetyTest`, `OperationsCanonicalUrlsTest`, Spec 078/144 operation tests, and browser monitoring smoke tests. | Lane artifacts: `*-latest.junit.xml` | OperationRun links and operations routes | `route-context-drift` | `route-parameter-repair` | no | yes | Focused operation URL tests, Spec 293 cutover lane, then affected lane rerun | grouped current root cause |
|
|
| Root cause group | Filament admin panel/resource URL context | Repeated failures call Filament `Resource::getUrl()` or `Page::getUrl()` without a current/admin panel and hit `hasTenancy()` on null; related table/action/form helpers also mount against null component state. Representative files include Findings authorization surfaces, Evidence snapshots, Backup/Restore bulk actions, Restore wizard tests, and browser Spec 174/177/192/198/277. | Lane artifacts: `*-latest.junit.xml` | Filament v5 admin panel URL generation and Livewire test setup | `panel-context-drift` | `panel-context-repair` | no | yes | Focused Filament tests, affected confidence/browser lane | grouped current root cause |
|
|
| Root cause group | Retired Filament route names | Confidence failures still expect removed `filament.tenant.*` resource routes or missing `filament.admin.resources.backup-schedules.index`; these must be rebaselined to current admin/workspace-first route truth without restoring TenantPanelProvider. | `./scripts/platform-test-lane confidence` | Filament cutover tests | `stale-test-expectation` | `test-rebaseline` | no | yes | Focused files, Spec 293 cutover lane | grouped current root cause |
|
|
| Root cause group | Legacy `/admin/t/...` path expectations | Confidence and browser failures still expect `/admin/t/{tenant}` after managed-environment cutover while the current path is `/admin/workspaces/{workspace}/environments/{environment}`. | Confidence and browser lane artifacts | Tenant cutover tests | `stale-test-expectation` | `test-rebaseline` | no | yes | Focused managed-environment and workspace overview tests, Spec 293 cutover lane | grouped current root cause |
|
|
| Root cause group | RBAC and deny-as-not-found status semantics | Fast-feedback/confidence failures show stale or unverified hidden/disabled/403/404/redirect semantics: expected 404/403 receiving 200, expected 404 receiving 302, capability/unit preflight booleans, tenant onboarding access, support handoff denial, and relation-manager action visibility. Owner policies/pages/actions must be read before deciding stale test vs runtime bug. | Fast-feedback/confidence/heavy-governance lane artifacts | RBAC, policies, action surfaces | `rbac-contract-drift` | `rbac-assertion-repair` | no | yes | Focused RBAC/action files and Spec 288 guard lane after any RBAC repair | grouped current root cause |
|
|
| Root cause group | Provider boundary and provider-operation start semantics | Fast-feedback/confidence unit and feature failures show provider boundary classification drift, `review_required` vs `blocked`, and provider-backed operations not dispatching expected jobs. This includes `ProviderBoundaryClassificationTest`, `ProviderBoundaryGuardrailTest`, `ProviderOperationStartGateTest`, onboarding provider start, and role definition sync. | Fast-feedback/confidence lane artifacts | Provider boundary and operation start services | `provider-boundary-drift` | `provider-fixture-or-contract-repair` | no | yes | ProviderConnections/Verification lane and focused provider unit tests | grouped current root cause |
|
|
| Root cause group | Browser smoke-login and UI copy/layout drift | Browser lane failures expect `Dashboard` after smoke-login, assert tenant dashboard spacing/copy, or expect `Manage memberships` on the tenant page. Screenshots are evidence-only while the lane is red. | `./scripts/platform-test-lane browser` | Browser smoke tests and UI copy/layout expectations | `browser-lane-drift` | `browser-expectation-repair` | no | yes | Focused browser tests and browser lane after route/panel repairs | grouped current root cause |
|
|
| Root cause group | Operation link helper and navigation contract drift | Fast-feedback/confidence failures show `OperationRunLinkContractGuard`, `RelatedNavigationResolverTest`, governance inbox links, and dashboard drill-through query-state expectations still asserting pre-workspace or pre-canonicalized URL/query semantics. | Fast-feedback/confidence lane artifacts | OperationRunLinks and navigation helpers | `route-context-drift` | `route-parameter-repair` | no | yes | Focused OperationRunLinks/navigation tests and Spec 293 lane | grouped current root cause |
|
|
| Root cause group | Unit fixture missing workspace tables | `RequiredPermissionsLinksTest` is a unit-context failure against `workspaces` table access while using in-memory SQLite without the needed schema/fixture. | Fast-feedback/confidence lane artifacts | Required permissions link unit fixtures | `missing-fixture` | `fixture-repair` | no | yes | Focused `tests/Unit/RequiredPermissionsLinksTest.php` | grouped current root cause |
|
|
| Root cause group | Summary and literal-count assertions | Confidence/heavy-governance show literal count drift in baseline compare query budget and tenant sync summary counts. The query budget likely needs a stable contract assertion; the tenant sync summary count must be owner-read before deciding stale assertion vs runtime counter bug. | Confidence/heavy-governance lane artifacts | Baseline performance guard and OpsUx summary counts | `stale-test-expectation` | `test-rebaseline` | no | yes | Focused baseline performance and tenant sync tests | grouped current root cause |
|
|
| Root cause group | Tenant membership and managed-environment role lifecycle drift | Confidence failures include last-owner guard copy/exception drift, membership audit setup exceptions, tenant diagnostics/route access 404s, and bootstrap snapshot drift. These sit under access-scope semantics and must not weaken least-privilege behavior. | `./scripts/platform-test-lane confidence` | Tenant RBAC and managed-environment memberships | `rbac-contract-drift` | `rbac-assertion-repair` | no | yes | Focused TenantRBAC files and Spec 288 guard lane | grouped current root cause |
|
|
| Guard lane | Spec 288 guard command | Current Spec 288 guard lane is green: 50 passed, 2055 assertions, 26.86s. No guard failures were added before broader repairs. | Spec 288 guard command from `spec.md` | workspace route, provider boundary, browser lane isolation, CI lane classification, no raw role strings | `fixed` | `no-fix-needed` | yes | no | Re-run after any RBAC/provider/route repair that touches guard ownership | green before broader repair |
|
|
| Guard lane | Spec 293 cutover regression command | Current Spec 293 cutover regression lane is green: 127 passed, 908 assertions, 72.56s. No cutover guard failures were added before broader repairs. | Spec 293 cutover command from `spec.md` | cutover navigation, no legacy tenant panel route restoration, operations links, provider connection navigation | `fixed` | `no-fix-needed` | yes | no | Re-run after any cutover/route/panel repair that touches guard ownership | green before broader repair |
|
|
| Guard lane | Spec 294 ProviderConnections/Verification command | Current ProviderConnections/Verification lane is green: 109 passed, 782 assertions, 64.87s. No provider guard failures were added before broader repairs. | `cd apps/platform && ./vendor/bin/sail artisan test --compact tests/Feature/ProviderConnections tests/Feature/Verification` | provider connection and verification semantics | `fixed` | `no-fix-needed` | yes | no | Re-run after any provider-boundary or verification repair | green before broader repair |
|
|
| Raw full suite | Baseline | Spec 295 observed raw suite red: 450 failed, 8 skipped, 4194 passed, 28831 assertions, 4686.08s. Spec 296 must re-run current baseline before fixing. | `cd apps/platform && ./vendor/bin/sail artisan test --compact` | suite governance | `route-context-drift` | `no-fix-needed` until current baseline rerun | no | yes | Re-run raw full suite during implementation | Seed from Spec 295; awaiting Spec 296 baseline |
|
|
| Lane split | fast-feedback | Spec 295 observed fast-feedback red with route context, panel context, authorization, RBAC/action, provider boundary, and monitoring/required-permissions failures. | `./scripts/platform-test-lane fast-feedback` | fast-feedback lane | `route-context-drift` | `no-fix-needed` until current baseline rerun | no | yes | Re-run fast-feedback if raw output is not classifiable | Seed from Spec 295 |
|
|
| Lane split | confidence | Spec 295 observed confidence red with route context, panel context, resource route, action helper, 404/403, and legacy URL drift. | `./scripts/platform-test-lane confidence` | confidence lane | `panel-context-drift` | `no-fix-needed` until current baseline rerun | no | yes | Re-run confidence if raw output is not classifiable | Seed from Spec 295 |
|
|
| Lane split | heavy-governance | Spec 295 observed heavy-governance red with operation route parameters, panel context, tenant sync summary count, and RBAC relation-manager UI drift. | `./scripts/platform-test-lane heavy-governance` | heavy-governance lane | `route-context-drift` | `no-fix-needed` until current baseline rerun | no | yes | `./scripts/platform-test-lane heavy-governance` | Seed from Spec 295 |
|
|
| Lane split | browser | Spec 295 observed browser red with smoke login, workspace operation route, Filament panel context, dashboard layout, old `/admin/t/...`, and tenant membership copy/action failures. | `./scripts/platform-test-lane browser` | browser lane | `browser-lane-drift` | `no-fix-needed` until current baseline rerun | no | yes | `./scripts/platform-test-lane browser` if browser files change | Seed from Spec 295 |
|
|
| Guard group | Legacy cutover route expectations | Known old `/admin/t/...`, TenantPanel, and `/admin/operations` assumptions must be re-checked and repaired without restoring runtime compatibility. | Spec 295 lane outputs | cutover tests | `stale-test-expectation` | `test-rebaseline` | no | yes | Spec 293 cutover lane | Seed root-cause group |
|
|
| Guard group | Workspace operation URL drift | Missing required `workspace` parameter for `admin.operations.index` or `admin.operations.view`. | Spec 295 raw/lane outputs | OperationRun links and operation page tests | `route-context-drift` | `route-parameter-repair` | no | yes | Focused OpsUx tests, Spec 293 lane, raw suite | Seed root-cause group |
|
|
| Guard group | Filament panel context drift | `Resource::getUrl()` or page rendering without current admin panel, including null `hasTenancy()` errors. | Spec 295 raw/lane outputs | Filament tests/resources | `panel-context-drift` | `panel-context-repair` | no | yes | Focused Filament files and affected lane | Seed root-cause group |
|
|
| Guard group | RBAC/action assertion drift | Hidden/disabled/403/404/action absent expectations may be stale or may reveal security bugs. | Spec 295 raw/lane outputs | Policies, resources, relation managers, action tests | `rbac-contract-drift` | `rbac-assertion-repair` | no | yes | Focused RBAC/action files and Spec 288 lane | Seed root-cause group |
|
|
| Guard group | Provider boundary residuals | `provider.capability_registry`, `review_required` vs `blocked`, dispatch count, and provider operation semantics need current proof. | Spec 295 raw/fast-feedback outputs | ProviderConnections and Verification | `provider-boundary-drift` | `provider-fixture-or-contract-repair` | no | yes | `cd apps/platform && ./vendor/bin/sail artisan test --compact tests/Feature/ProviderConnections tests/Feature/Verification` | Seed root-cause group |
|
|
| Guard group | Browser screenshot/copy/path drift | Browser tests may generate evidence screenshots and stale path/copy failures. | Spec 295 browser output | browser lane | `browser-lane-drift` | `browser-expectation-repair` | no | yes | `./scripts/platform-test-lane browser` | Seed root-cause group |
|
|
|
|
## Implementation Notes
|
|
|
|
- Add current Spec 296 baseline rows above or below the seed rows.
|
|
- Current raw full-suite output was too broad/truncated in the tool result; use lane split rows as the authoritative repair inventory before touching tests/runtime.
|
|
- When a group is fixed, update `Classification` to `fixed` only if the validation command proves it.
|
|
- Do not delete seed rows; mark them superseded by current baseline rows if current output changes.
|
|
- Every changed file must also appear in `fix-log.md`.
|