ahmido
439248ba15
Implements the 074 verification checklist framework. Highlights: - Versioned verification report contract stored in operation_runs.context.verification_report (DB-only viewer). - Strict sanitizer/redaction (evidence pointers only; no tokens/headers/payloads) + schema validation. - Centralized BADGE-001 semantics for check status, severity, and overall report outcome. - Deterministic start (dedupe while active) via shared StartVerification service; capability-first authorization (non-member 404, member missing capability 403). - Completion audit event (verification.completed) with redacted metadata. - Integrations: OperationRun detail viewer, onboarding wizard verification step, provider connection start surfaces. Tests: - vendor/bin/sail artisan test --compact tests/Feature/Verification tests/Unit/Badges/VerificationBadgesTest.php - vendor/bin/sail bin pint --dirty Co-authored-by: Ahmed Darrazi <ahmeddarrazi@MacBookPro.fritz.box> Reviewed-on: #89
187 lines
14 KiB
Markdown
187 lines
14 KiB
Markdown
# Feature Specification: Verification Checklist Framework (Enterprise-Ready)
|
||
|
||
**Feature Branch**: `074-verification-checklist`
|
||
**Created**: 2026-02-03
|
||
**Status**: Draft
|
||
**Input**: User description: "Replace binary verification UX with a structured, reusable verification checklist attached to verification runs; DB-only viewing; enterprise semantics (reason codes, audit, idempotency, RBAC)."
|
||
|
||
## Clarifications
|
||
|
||
### Session 2026-02-03
|
||
|
||
- Q: What idempotency policy do we want for “Start verification”? → A: Dedupe only while a run is active (queued/running); once completed/failed, “Start verification” creates a new run.
|
||
- Q: Who should be allowed to view verification reports? → A: Any authenticated workspace member with access to the tenant scope may view reports; starting verification requires a separate capability.
|
||
- Q: What policy should we use for `reason_code` taxonomy? → A: Versioned central taxonomy with a small baseline set + reserved `ext.*` namespace for feature-specific extensions.
|
||
- Q: What’s the required evidence/redaction policy for `evidence` in check results? → A: Evidence is strictly structured safe pointers only (internal IDs, masked strings, hashes); never raw payloads, tokens, claims, headers, or full error bodies.
|
||
- Q: Should “Next steps” CTAs be links only, or can they trigger server-side actions? → A: Links only (navigation-only) in v1.
|
||
|
||
## User Scenarios & Testing *(mandatory)*
|
||
|
||
<!--
|
||
IMPORTANT: User stories should be PRIORITIZED as user journeys ordered by importance.
|
||
Each user story/journey must be INDEPENDENTLY TESTABLE - meaning if you implement just ONE of them,
|
||
you should still have a viable MVP (Minimum Viable Product) that delivers value.
|
||
|
||
Assign priorities (P1, P2, P3, etc.) to each story, where P1 is the most critical.
|
||
Think of each story as a standalone slice of functionality that can be:
|
||
- Developed independently
|
||
- Tested independently
|
||
- Deployed independently
|
||
- Demonstrated to users independently
|
||
-->
|
||
|
||
### User Story 1 - Operator sees what’s wrong (Priority: P1)
|
||
|
||
As a workspace member onboarding or operating a managed tenant, I can run “Verify access” and see a structured checklist that clearly shows which checks passed, which failed, and what to do next.
|
||
|
||
**Why this priority**: This is the primary value of verification: reduce ambiguity and enable fast, correct remediation.
|
||
|
||
**Independent Test**: Seed a verification run with a report containing mixed outcomes and confirm the viewer renders an accurate summary, per-check status, and next steps without making any external calls.
|
||
|
||
**Acceptance Scenarios**:
|
||
|
||
1. **Given** a completed verification run with 2 failed checks, **When** I open the verification report viewer, **Then** I see an overall summary (“Needs attention” or “Blocked”), counts, and the two failed checks with actionable next steps.
|
||
2. **Given** a verification run that is still in progress, **When** I open the viewer, **Then** I see a “Running” state and partial results (if available) without errors.
|
||
|
||
---
|
||
|
||
### User Story 2 - Deterministic starts (idempotency / dedupe) (Priority: P2)
|
||
|
||
As an operator, if I click “Start verification” multiple times for the same tenant + provider connection + flow, the system behaves deterministically: it does not start duplicate active runs and guides me to the already-running run.
|
||
|
||
**Why this priority**: Prevents confusing duplicates, reduces load, and makes support/debugging repeatable.
|
||
|
||
**Independent Test**: Attempt to start verification twice for the same identity and assert that only one active run exists and the UI returns a consistent “already running” outcome.
|
||
|
||
**Acceptance Scenarios**:
|
||
|
||
1. **Given** an active verification run exists for the same identity, **When** I click “Start verification”, **Then** no duplicate run is started and I am directed to view the active run/report.
|
||
2. **Given** no active verification run exists (including when the most recent run is completed or failed), **When** I click “Start verification”, **Then** a new run starts and I can view its report as it progresses.
|
||
|
||
---
|
||
|
||
### User Story 3 - Least-privilege and safe disclosure (Priority: P3)
|
||
|
||
As a workspace member with access to the tenant scope (including read-only), I can view verification reports but cannot start verification unless I have the start capability. As a non-member, I cannot discover that a tenant or report exists.
|
||
|
||
**Why this priority**: Verification data can leak operational posture; access must follow least-privilege and “deny-as-not-found” for non-members.
|
||
|
||
**Independent Test**: Validate both authorization paths: read-only can view but cannot start; non-member receives a not-found response for all tenant-scoped verification routes.
|
||
|
||
**Acceptance Scenarios**:
|
||
|
||
1. **Given** I am a workspace member without the “start verification” capability, **When** I open the verification page, **Then** I can view past reports but the “Start verification” action is disabled and cannot be executed.
|
||
2. **Given** I am not a member of the workspace/tenant scope, **When** I attempt to access the verification report route, **Then** I receive a not-found response with no identifying hints.
|
||
|
||
---
|
||
|
||
### Edge Cases
|
||
|
||
- Report missing or malformed (e.g., run exists but report is absent or partial) → viewer shows a safe “Report unavailable” state and guidance.
|
||
- Unknown check keys or unknown reason codes (newer schema written by a newer verifier) → viewer degrades gracefully, still showing status/message/next steps when present.
|
||
- Large reports (near upper bound, e.g., 50 checks) → viewer remains responsive and summary counts remain correct.
|
||
- A run transitions from running → complete while the user is viewing → the viewer refreshes safely or the user can re-open without inconsistent states.
|
||
- Evidence contains unexpected fields → redaction rules prevent sensitive values from being displayed.
|
||
|
||
## Out of Scope
|
||
|
||
- Introducing a separate monitoring/observability platform beyond the existing run tracking and audit log.
|
||
- Any workflow that requires client-side handling of secrets.
|
||
- A full overhaul of onboarding wizards beyond replacing/embedding verification status with the checklist viewer.
|
||
- Provider job orchestration redesign unrelated to running verification checks.
|
||
- Server-side actions triggered directly from the checklist viewer (v1 is navigation-only).
|
||
|
||
## Requirements *(mandatory)*
|
||
|
||
**Constitution alignment (required):** If this feature introduces any external provider calls, any write/change behavior,
|
||
or any long-running/background work, the spec MUST describe safety gates (preview/confirmation/audit), tenant isolation,
|
||
run observability (run identity, visibility, and outcomes), and tests. If security-relevant DB-only actions intentionally
|
||
skip run tracking, the spec MUST describe the audit log entries.
|
||
|
||
**Constitution alignment (RBAC-UX):** If this feature introduces or changes authorization behavior, the spec MUST:
|
||
- state which authorization plane(s) are involved (tenant-scoped admin area vs platform/system admin area),
|
||
- ensure any cross-plane access is deny-as-not-found (404),
|
||
- explicitly define 404 vs 403 semantics:
|
||
- non-member / not entitled to tenant scope → 404 (deny-as-not-found)
|
||
- member but missing capability → 403
|
||
- describe how authorization is enforced server-side (Gates/Policies) for every mutation/operation-start/credential change,
|
||
- reference the canonical capability registry (no raw capability strings; no role-string checks in feature code),
|
||
- ensure global search is tenant-scoped and non-member-safe (no hints; inaccessible results treated as 404 semantics),
|
||
- ensure destructive-like actions require explicit user confirmation,
|
||
- include at least one positive and one negative authorization test, and note any RBAC regression tests added/updated.
|
||
|
||
**Constitution alignment (OPS-EX-AUTH-001):** OIDC/SAML login handshakes may perform synchronous outbound HTTP (e.g., token exchange)
|
||
on `/auth/*` endpoints without an `OperationRun`. This MUST NOT be used for Monitoring/Operations pages.
|
||
|
||
**Constitution alignment (BADGE-001):** If this feature changes status-like badges (status/outcome/severity/risk/availability/boolean),
|
||
the spec MUST describe how badge semantics stay centralized (no ad-hoc mappings) and which tests cover any new/changed values.
|
||
|
||
### Functional Requirements
|
||
|
||
- **FR-001 — Canonical verification report contract**: The system MUST generate a versioned “Verification Report” document for each verification run, including: schema version, flow identifier, identity/scope, generated timestamp, summary counts, and a list of check results.
|
||
|
||
- **FR-002 — Check result contract**: Each check result MUST include: stable key, title, status (pass/fail/warn/skip/running), severity (info/low/medium/high/critical), blocking flag, reason code, human-readable message, safe evidence pointers, and one or more “next steps” actions (where applicable).
|
||
|
||
Evidence MUST be strictly limited to structured safe pointers (internal IDs, masked strings, hashes) and MUST NOT contain raw payloads, tokens, claims, headers, or full error bodies.
|
||
|
||
Next steps in v1 MUST be navigation-only (links to internal pages or external documentation) and MUST NOT trigger server-side actions.
|
||
|
||
- **FR-003 — Stable reason code taxonomy**: The system MUST use stable, documented reason codes for failed/warned/skipped outcomes so that support, automation, and future UI changes remain consistent.
|
||
|
||
The taxonomy MUST include a small baseline set of cross-cutting codes and MUST reserve an `ext.*` namespace for flow-specific or check-specific extensions.
|
||
|
||
- **FR-004 — DB-only viewing**: Viewing a verification checklist MUST be read-only and MUST NOT trigger any external calls (e.g., no provider API calls, no HTTP calls, no background jobs started as a side effect of rendering).
|
||
|
||
- **FR-005 — Start verification creates a run**: Starting verification MUST create (or reuse, per dedupe policy) a new verification run record and begin executing the verification checks using existing background processing.
|
||
|
||
- **FR-006 — Dedupe / idempotency**: If a verification run is already active for the same identity (tenant + provider connection + flow), the system MUST NOT start a duplicate active run; it MUST present a clear “already running” outcome and an affordance to view the active run/report.
|
||
|
||
If no run is active (including when the most recent run is completed or failed), “Start verification” MUST create a new run.
|
||
|
||
- **FR-007 — Capability-first authorization**: Permission checks for viewing and starting verification MUST reference the canonical capability registry (no string-literal capability checks in feature code).
|
||
|
||
- **FR-008 — RBAC UX semantics**: Non-members attempting to access tenant-scoped verification pages/routes MUST receive not-found responses. Members lacking the “start” capability MUST be able to view reports but MUST NOT be able to start verification (UI disabled + server-side enforcement).
|
||
|
||
Viewing reports MUST NOT require the start capability.
|
||
|
||
- **FR-009 — Standardized UI semantics**: The viewer MUST render consistent status labels, a summary banner (e.g., Ready / Needs attention / Blocked), and per-check expandable details with standardized “Next steps” calls-to-action.
|
||
|
||
- **FR-010 — Reuse across suite**: The framework MUST be adoptable by multiple verification flows without re-implementing viewer logic, including: managed tenant onboarding verification, provider connection verification, RBAC setup verification, consent & permission verification, and future readiness/health checks.
|
||
|
||
- **FR-011 — Auditing**: Starting and completing verification MUST emit audit events with stable action identifiers and redaction rules, recording minimal metadata (workspace/tenant identifiers, run identifier, and result counts).
|
||
|
||
### Key Entities *(include if feature involves data)*
|
||
|
||
- **Verification Flow**: A named verification context (e.g., managed tenant onboarding) that defines which checks run.
|
||
- **Verification Identity (Scope)**: The set of identifiers that uniquely represent “what is being verified” (tenant + provider connection + flow).
|
||
- **Verification Run**: A single execution attempt for a given identity that produces a report (and is auditable).
|
||
- **Verification Report**: A versioned, structured document attached to a run, containing summary and check results.
|
||
- **Check Definition**: A reusable definition of an atomic readiness check (key, title, expected preconditions, severity, blocking behavior).
|
||
- **Check Result**: The outcome of executing a check within a report.
|
||
- **Reason Code**: A stable, machine-readable classification of why a check is pass/fail/warn/skip.
|
||
- **Next Step**: An actionable remediation hint (label + optional destination/action) that helps the operator resolve a failed check.
|
||
- **Evidence Pointer**: Safe references that support diagnostics (IDs, masked strings, hashes), without exposing secrets.
|
||
|
||
### Assumptions
|
||
|
||
- A run-tracking mechanism already exists and can store an attached, versioned verification report per run.
|
||
- A canonical capability registry exists and is the source of truth for permission checks.
|
||
- An audit logging mechanism exists that can record start/complete events with redaction.
|
||
- Verification execution uses existing background processing patterns (no new observability platform is introduced).
|
||
|
||
### Dependencies
|
||
|
||
- Workspace membership and tenant-scoped authorization boundaries are already modeled.
|
||
- Run visibility rules support “deny-as-not-found” behavior for non-members.
|
||
- UI surfaces exist (or can be added) where “Next steps” can route users for remediation.
|
||
|
||
## Success Criteria *(mandatory)*
|
||
|
||
### Measurable Outcomes
|
||
|
||
- **SC-001 (Clarity)**: In a usability test with a pre-seeded failed report, 90% of operators can identify the top blocking failure and the recommended next step within 60 seconds.
|
||
- **SC-002 (Determinism)**: When “Start verification” is triggered repeatedly for the same identity while a run is active, the system starts at most 1 active run (0 duplicates) and always provides a path to view the active run.
|
||
- **SC-003 (Safety / data minimization)**: Verification reports contain no secrets or tokens; evidence is limited to safe pointers (validated by automated tests and/or static checks).
|
||
- **SC-004 (Performance)**: The verification report viewer renders within 200ms server time for a typical report of up to 50 checks.
|
||
- **SC-005 (Authorization)**: Non-member access to tenant-scoped verification pages results in not-found responses in 100% of tested cases; members without the start capability cannot execute start actions in 100% of tested cases.
|