ahmido
3c38192405
## Summary This PR implements Spec 206 end to end and establishes the first checked-in test suite governance foundation for the platform app. Key changes: - add manifest-backed test lanes for fast-feedback, confidence, browser, heavy-governance, profiling, and junit - add budget and report helpers plus app-local artifact generation under `apps/platform/storage/logs/test-lanes` - add repo-root Sail-friendly lane/report wrappers - switch the default contributor test path to the fast-feedback lane - introduce explicit fixture profiles and cheaper defaults for shared tenant/provider test setup - add minimal/heavy factory states for tenant and provider connection setup - migrate the first high-usage and provider-sensitive tests to explicit fixture profiles - document budgets, taxonomy rules, DB reset guidance, and the full Spec 206 plan/contracts/tasks set ## Validation Executed during implementation: - focused Spec 206 guard/support/factory validation pack: 31 passed - provider-sensitive regression pack: 29 passed - first high-usage caller migration pack: 120 passed - lane routing and wrapper validation succeeded - pint completed successfully Measured lane baselines captured in docs: - fast-feedback: 176.74s - confidence: 394.38s - heavy-governance: 83.66s - browser: 128.87s - junit: 380.14s - profiling: 2701.51s - full-suite baseline anchor: 2624.60s ## Notes - Livewire v4 / Filament v5 runtime behavior is unchanged by this PR. - No new runtime routes, product UI flows, or database migrations are introduced. - Panel provider registration remains unchanged in `bootstrap/providers.php`. Co-authored-by: Ahmed Darrazi <ahmed.darrazi@live.de> Reviewed-on: #239
238 lines
10 KiB
Markdown
238 lines
10 KiB
Markdown
# Data Model: Test Suite Governance & Performance Foundation
|
|
|
|
This feature does not introduce new runtime database tables. The data-model work formalizes repository-level governance objects that define how tests are grouped, how heavy setup is declared, and how runtime drift is reported. The first slice uses four operational lanes (`fast-feedback`, `confidence`, `browser`, and `heavy-governance`) plus two support-lane entries (`profiling` and `junit`) that share the same manifest and reporting contract shape while keeping narrower responsibilities. Field names in this document are conceptual and may appear in camelCase form in the checked-in manifest and logical reporting contracts. In particular, `lane_id` maps to manifest `id` and report `laneId`, `artifact_modes` maps to manifest `artifacts`, accepted-run `artifactModes`, and report `artifacts[].artifactMode`, and `enforcement_level` maps to manifest-contract `enforcement`.
|
|
|
|
## 1. Test Lane
|
|
|
|
### Purpose
|
|
|
|
Represents one checked-in execution path or support-lane entry for the suite.
|
|
|
|
### Fields
|
|
|
|
- `lane_id`: stable identifier such as `fast-feedback`, `confidence`, `heavy-governance`, `browser`, `profiling`, or `junit`
|
|
- `governance_class`: `fast`, `confidence`, `heavy`, or `support`
|
|
- `description`: contributor-facing statement of the lane's purpose
|
|
- `intended_audience`: the contributor or reviewer audience the lane is optimized for
|
|
- `included_families`: written list of the families the lane intentionally owns
|
|
- `excluded_families`: written list of the families the lane intentionally leaves out
|
|
- `ownership_expectations`: contributor-facing statement of when the lane is expected to be run and maintained
|
|
- `default_entry_point`: boolean
|
|
- `parallel_mode`: `required`, `optional`, or `forbidden`
|
|
- `selectors`: include and exclude rules for suites, directories, files, and groups
|
|
- `artifact_modes`: list of expected outputs such as `summary`, `junit-xml`, `profile-top`, or `budget-report`
|
|
- `budget`: runtime target and enforcement level
|
|
- `db_strategy`: expected connection and reset discipline
|
|
|
|
### Validation rules
|
|
|
|
- Exactly one lane may be the default contributor entry point.
|
|
- The first-slice manifest must include the four operational lanes and the two support runs.
|
|
- Support runs remain lane-shaped manifest entries with `governance_class = support` and the same reporting envelope as operational lanes.
|
|
- A lane that emits `profile-top` must use `parallel_mode = forbidden`.
|
|
- A lane with `lane_id = browser` must not be the default contributor lane.
|
|
- A lane must have at least one positive selector.
|
|
- Every lane declaration must record its purpose or description, intended audience, included families, excluded families, and ownership expectations.
|
|
- The `fast-feedback` lane budget must carry `baseline_delta_target_percent = 50` for the first slice.
|
|
|
|
## 2. Lane Selector
|
|
|
|
### Purpose
|
|
|
|
Represents one inclusion or exclusion rule that determines lane membership.
|
|
|
|
In the first-slice manifest contract, selector records are serialized into the flattened include or exclude arrays on each lane entry. The richer selector fields below remain conceptual metadata for planning, review, and future helper logic rather than required manifest fields.
|
|
|
|
### Fields
|
|
|
|
- `selector_type`: `testsuite`, `path`, `group`, or `file`
|
|
- `selector_value`: exact suite name, directory path, group name, or file path
|
|
- `inclusion_mode`: `include` or `exclude`
|
|
- `rationale`: short explanation of why the selector belongs to the lane
|
|
- `cost_class`: optional `normal`, `heavy`, or `special-case`
|
|
|
|
### Relationships
|
|
|
|
- Many selectors belong to one test lane.
|
|
- Selectors may be shared conceptually across lanes but are evaluated per lane declaration.
|
|
|
|
## 3. Test Classification Rule
|
|
|
|
### Purpose
|
|
|
|
Defines what makes a test legitimately belong to a particular classification.
|
|
|
|
### Fields
|
|
|
|
- `classification`: `unit`, `feature-integration`, `browser`, or `architecture-governance`
|
|
- `laravel_boot_allowed`: boolean
|
|
- `database_allowed`: boolean
|
|
- `browser_runtime_required`: boolean
|
|
- `livewire_or_filament_mount_allowed`: boolean
|
|
- `default_lane_target`: lane identifier
|
|
- `promotion_conditions`: list of conditions that force escalation into a heavier lane
|
|
|
|
### Validation rules
|
|
|
|
- `unit` classification must not require browser runtime.
|
|
- `browser` classification must require browser runtime.
|
|
- Tests that need broad file scans, whole-surface discovery, or smoke assertions across many pages must not default into the fast-feedback lane.
|
|
|
|
## 4. Shared Fixture Profile
|
|
|
|
### Purpose
|
|
|
|
Represents a named setup profile for a shared test helper.
|
|
|
|
### Fields
|
|
|
|
- `helper_name`: example `createUserWithTenant`
|
|
- `profile_name`: `minimal`, `full`, or another explicit cost-bearing name
|
|
- `default_profile`: boolean
|
|
- `creates_workspace`: boolean
|
|
- `creates_membership`: boolean
|
|
- `creates_credentials`: boolean
|
|
- `creates_provider_connection`: boolean
|
|
- `sets_session_context`: boolean
|
|
- `creates_ui_context`: boolean
|
|
- `clears_capability_caches`: boolean
|
|
- `notes`: contributor guidance on when this profile is appropriate
|
|
|
|
### Validation rules
|
|
|
|
- Each shared helper must have exactly one default profile.
|
|
- A default profile should not create workspace, membership, credentials, provider connection state, session context, cache-clearing side effects, or UI context unless the helper's primary job is specifically to set up that heavier behavior.
|
|
- Profiles with materially heavier behavior must use explicit names rather than boolean flags only.
|
|
|
|
## 5. Factory Cost Profile
|
|
|
|
### Purpose
|
|
|
|
Describes the cost posture of a factory state used in tests.
|
|
|
|
### Fields
|
|
|
|
- `factory_name`
|
|
- `state_name`
|
|
- `cost_class`: `minimal`, `standard`, or `heavy`
|
|
- `creates_relationships`: list of related models or graph expansions
|
|
- `intended_usage`: short explanation of the state's purpose
|
|
- `safe_default`: boolean
|
|
|
|
### Validation rules
|
|
|
|
- A heavy state must document which additional relationships it creates.
|
|
- Factories used in high-volume tests should expose at least one minimal or standard state that avoids surprising graph expansion.
|
|
|
|
## 6. Database Reset Policy
|
|
|
|
### Purpose
|
|
|
|
Formalizes the allowed DB behavior for a suite or lane.
|
|
|
|
### Fields
|
|
|
|
- `target_scope`: `unit`, `feature`, `browser`, `pgsql`, or `lane-specific`
|
|
- `connection_mode`: `sqlite-memory`, `pgsql`, or `mixed`
|
|
- `reset_strategy`: `none`, `refresh-database`, or `dedicated-suite`
|
|
- `seeds_policy`: `forbidden`, `restricted`, or `allowed-by-exception`
|
|
- `schema_baseline_candidate`: boolean
|
|
- `notes`
|
|
|
|
### Validation rules
|
|
|
|
- A policy using `connection_mode = sqlite-memory` should treat schema-baseline work as optional evidence rather than assumed default value.
|
|
- A policy using `reset_strategy = dedicated-suite` must name the target suite or command path that justifies the exception.
|
|
|
|
## 7. Runtime Budget
|
|
|
|
### Purpose
|
|
|
|
Represents the expected wall-clock target for a lane or known heavy family.
|
|
|
|
### Fields
|
|
|
|
- `budget_id`
|
|
- `target_type`: `lane` or `family`
|
|
- `target_id`: lane identifier or family identifier
|
|
- `threshold_seconds`
|
|
- `baseline_source`: `measured-current-suite` or `measured-lane`
|
|
- `enforcement_level`: `report-only`, `warn`, or `hard-fail`
|
|
- `baseline_delta_target_percent`: optional percentage reduction target versus the current full-suite baseline when applicable, such as the fast-feedback 50% target
|
|
- `lifecycle_state`: `draft`, `measured`, `documented`, or `enforced`
|
|
- `review_cadence`: short rule such as `tighten after two stable runs`
|
|
|
|
### State transitions
|
|
|
|
- `draft` → `measured`
|
|
- `measured` → `documented`
|
|
- `documented` → `enforced`
|
|
|
|
## 8. Lane Report Artifact
|
|
|
|
### Purpose
|
|
|
|
Represents one machine-readable or human-readable artifact emitted by a lane.
|
|
|
|
### Fields
|
|
|
|
- `artifact_type`: `summary`, `junit-xml`, `profile-top`, or `budget-report`
|
|
- `relative_path`: app-root relative path under `storage/logs/test-lanes`
|
|
- `machine_readable`: boolean
|
|
- `generated_by_lanes`: list of lane identifiers
|
|
- `retention_scope`: `local-ephemeral`
|
|
|
|
### Validation rules
|
|
|
|
- Artifact paths must be relative to `apps/platform` because `scripts/platform-sail` changes into the app directory.
|
|
- Machine-readable budget evaluation must include the measured duration, threshold, and status.
|
|
- First-slice artifact retention is local-only under `storage/logs/test-lanes`; CI upload is later hardening work, not part of this contract.
|
|
|
|
## 8a. Family Budget Threshold
|
|
|
|
### Purpose
|
|
|
|
Represents the budget contract for a heavy file, group, path, or suite family that is tracked alongside lane-level budgets.
|
|
|
|
### Fields
|
|
|
|
- `family_id`: stable identifier for the heavy family
|
|
- `selector_type`: `testsuite`, `path`, `group`, or `file`
|
|
- `selectors`: one or more selectors that identify the tracked family
|
|
- `threshold_seconds`: allowed wall-clock target for the family cluster
|
|
- `baseline_source`: `measured-current-suite` or `measured-lane`
|
|
- `enforcement_level`: `report-only`, `warn`, or `hard-fail`
|
|
- `notes`
|
|
- `lifecycle_state`: `draft`, `measured`, `documented`, or `enforced`
|
|
|
|
### Validation rules
|
|
|
|
- Every first-slice manifest must carry at least one family budget threshold in addition to lane-level budgets.
|
|
- Family budget thresholds must use the same baseline-backed source vocabulary as lane budgets.
|
|
|
|
## 9. First-Slice Governance Inventory
|
|
|
|
### Current suite footprint
|
|
|
|
- Approximate test file count: `1,122`
|
|
- Feature files: `873`
|
|
- Unit files: `234`
|
|
- Browser files: `12`
|
|
- Architecture files: `2`
|
|
- Deprecation files: `1`
|
|
|
|
### Current hotspots
|
|
|
|
- `createUserWithTenant()` is referenced by roughly `607` test files and currently provisions user, tenant, workspace, workspace membership, session context, capability cache clears, and provider connection state by default.
|
|
- `tests/Pest.php` applies `RefreshDatabase` broadly to `tests/Feature` and `tests/Browser`.
|
|
- Existing explicit Pest groups are sparse, with `ops-ux` as the largest visible group seam.
|
|
|
|
### First-slice target objects
|
|
|
|
- `fast-feedback` lane definition
|
|
- `confidence` lane definition
|
|
- `heavy-governance` lane definition
|
|
- `browser` execution target
|
|
- `profiling` and `junit` support targets
|
|
- `createUserWithTenant` minimal and full fixture profiles
|
|
- At least one additional factory or fixture cluster with explicit minimal and heavy cost profiles |