TenantAtlas/specs/113-platform-ops-runbooks/spec.md

Feature Specification: Platform Ops Runbooks (Operator Control Plane) for Backfills & Data Repair

Feature Branch: [113-platform-ops-runbooks]
Created: 2026-02-26
Status: Draft
Input: Operator control plane runbooks for safe backfills and data repair; deploy-time automatic execution; operator re-run via /system; never exposed in customer UI.

Clarifications

Session 2026-02-26

• Q: /system Session Isolation Strategy (v1) → A: B — Use a distinct session cookie name/config for /system.
• Q: OperationRun.type for the findings lifecycle backfill runbook → A: Use findings.lifecycle.backfill (consistent with the operation catalog). Runbook trigger is exclusive to /system; any /admin trigger is removed / feature-flagged off.
• Q: v1 scope selector for running the runbook → A: All tenants (default) + Single tenant (picker).
• Q: Failure notification delivery (v1) → A: Deliver via existing alert destinations (Teams webhook / Email) when configured, and always notify the initiating platform operator in-app.
• Q: /system/login rate limiting policy (v1) → A: 10/min per IP + username (combined key).
• Q: Platform “allowed tenant universe” (v1) → A: All non-platform tenants present in the database (tenants.external_id != 'platform'). The System UI must not allow selecting or targeting the platform tenant.

Spec Scope Fields (mandatory)

• Scope: canonical-view (platform control plane)
• Primary Routes:
  • /system/ops/runbooks (runbook catalog + preflight + run)
  • /system/ops/runs (run history + run details)
  • /admin/* (explicitly remove any maintenance/backfill affordances)
• Data Ownership:
  • Tenant-owned customer data that may be modified by runbooks (e.g., “findings” lifecycle/workflow fields)
  • Platform-owned operational records (operation runs, audit events, operator notifications)
• RBAC:
  • Platform identity only (separate from tenant users)
  • Capabilities (v1 minimum): platform.ops.view, platform.runbooks.view, platform.runbooks.run
  • Optional granular capability for this runbook: platform.runbooks.findings.lifecycle_backfill

For canonical-view specs, the spec MUST define:

• Default filter behavior when tenant-context is active: the runbook defaults to All tenants scope; if a tenant is explicitly selected, all counts/changes MUST be limited to that tenant only.
• Explicit entitlement checks preventing cross-tenant leakage: a tenant-context user MUST NOT be able to access /system/* (deny-as-not-found). Platform operators MUST only be able to target tenants within the platform’s allowed tenant universe.

User Scenarios & Testing (mandatory)

User Story 1 - Operator runs a runbook safely (Priority: P1)

As a platform operator, I can run a predefined “Rebuild Findings Lifecycle” runbook from /system with a clear preflight, explicit confirmation, and an audited, trackable run record.

Why this priority: This is the primary operator workflow that eliminates the need for SSH/manual scripts and reduces risk for customer-impacting data changes.

Independent Test: Fully testable by visiting /system/ops/runbooks, running preflight, starting a run, and verifying the run record + audit events exist.

Acceptance Scenarios:

1. Given an authorized platform operator, When they open /system/ops/runbooks, Then they see the runbook catalog including “Rebuild Findings Lifecycle” and an operator warning that actions may modify customer data.
2. Given preflight reports affected_count > 0, When the operator confirms the run, Then a new operation run is created and the UI links to “View run”.
3. Given preflight reports affected_count = 0, When the operator attempts to run, Then the run action is disabled with a clear “Nothing to do” explanation.
4. Given the operator chooses “All tenants”, When they confirm, Then typed confirmation is required (e.g., entering BACKFILL) and a reason is required.

---

User Story 2 - Customers never see maintenance actions (Priority: P1)

As a tenant (customer) user, I never see backfill/repair buttons and cannot access the operator control plane.

Why this priority: Exposing maintenance controls in customer UI is an enterprise anti-pattern and undermines product trust.

Independent Test: Fully testable by checking /admin UI surfaces and attempting direct navigation to /system/* as a tenant user.

Acceptance Scenarios:

1. Given a tenant user session, When the user requests /system/ops/runbooks, Then the response is 404 (deny-as-not-found).
2. Given production-like configuration, When a tenant user views relevant /admin screens, Then there is no maintenance/backfill/repair UI.

---

User Story 3 - Same logic for deploy-time and operator re-run (Priority: P2)

As a platform operator and as a deploy pipeline, the same runbook logic can be executed consistently so that deploy-time backfills are automatic, and manual re-runs remain available and safe.

Why this priority: A single execution path reduces drift between “what deploy does” and “what operators can re-run”, and improves reliability.

Independent Test: Fully testable by running the operation twice and verifying idempotency and consistent preflight/run results for the same scope.

Acceptance Scenarios:

1. Given the runbook was executed once successfully, When it is executed again with the same scope, Then the second run reports updated_count = 0 (idempotent behavior).

Edge Cases

• Lock already held: another run is in-progress for the same scope (All tenants or the same tenant).
• Large dataset: preflight must remain fast enough for operator use; writes must be chunked to avoid long locks.
• Partial failure: some tenants/records fail while others succeed; run outcome and audit still record what happened.
• Missing reason: an All-tenants or break-glass run cannot start without a reason.

Requirements (mandatory)

Constitution alignment notes

• No customer-plane maintenance: Any maintenance/backfill/repair affordance in /admin is explicitly out of scope for customer UX.
• Run observability: Customer-impacting writes MUST be executed as a tracked operation run with clear status/outcome and operator-facing surfaces.
• Safety gates: Preflight → explicit confirmation → audited execution is mandatory.

Functional Requirements

• FR-001 (Remove Customer Exposure): The system MUST not expose any backfill/repair controls in /admin in production-like environments. Any legacy /admin trigger for the findings lifecycle backfill MUST be removed or disabled (feature-flag off by default).
• FR-002 (Runbook Catalog): The system MUST provide a /system/ops/runbooks catalog listing predefined runbooks and their descriptions.
• FR-003 (Runbook: Rebuild Findings Lifecycle): The system MUST provide a runbook that supports:
  • Preflight (read-only) showing at least affected_count.
  • Run (write) that starts a tracked operation run and links to “View run”.
  • Scope selection: All tenants (default) and Single tenant (picker).
  • Safe confirmation: includes scope + preflight count + “modifies customer data” warning.
  • Typed confirmation for All-tenants scope (e.g., BACKFILL).
  • Run disabled when preflight indicates nothing to do.
• FR-004 (Single Source of Truth): The system MUST implement the runbook logic once and reuse it across:
  • deploy-time execution (automation)
  • operator UI execution in /system The two paths MUST produce consistent results for the same scope.
• FR-005 (Operation Run Tracking): Each run MUST create a run record including:
  • run type identifier: findings.lifecycle.backfill
  • scope (all tenants vs single tenant)
  • actor (platform user, including break-glass marker when applicable)
  • outcome/status transitions owned by the service layer
  • numeric summary counts using a centralized allow-list of keys
  • run context containing: preflight.affected_count, updated_count, skipped_count, error_count, and duration
• FR-006 (Audit Events): The system MUST write audit events for start, completion, and failure. Audit writing MUST be fail-safe (audit failures do not crash the operation run).
• FR-007 (Reasons for Sensitive Runs): All-tenants runs and break-glass runs MUST require a reason:
  • reason_code: one of DATA_REPAIR, INCIDENT, SUPPORT, SECURITY
  • reason_text: free text (max 500 characters)
• FR-008 (Locking & Idempotency): The system MUST prevent concurrent runs for the same scope via locking and MUST be idempotent (a second execution does not re-write already-correct data).
• FR-009 (Operator Notification on Failure): A failed run MUST notify operator targets with run type + scope + a link to “View run”. v1 delivery:
  • If alert destinations are configured, deliver via existing destinations (Teams webhook / Email).
  • Always notify the initiating platform operator in-app. Success notifications are optional and SHOULD be off by default.

Security & Non-Functional Requirements

• SR-001 (Control Plane Isolation): /system MUST be isolated to platform identity and MUST deny tenant-plane access as 404 (anti-enumeration).
• SR-002 (404 vs 403 Semantics):
  • Non-platform users or wrong plane → 404
  • Platform user lacking required capability → 403
• SR-003 (Login Throttling): The /system/login surface MUST be rate limited at 10/min per IP + username (combined key) and failed login attempts MUST be audited.
• SR-004 (Session Isolation Strategy): v1 MUST isolate control plane sessions from tenant sessions by using a distinct session cookie name/config for /system (same domain). A dedicated subdomain with separate cookie scope may be introduced later.
• SR-005 (Break-glass Visibility & Enforcement): Break-glass mode MUST be visually obvious and MUST require a reason; break-glass usage MUST be recorded on the run and in audit.
• NFR-001 (Performance & Safety):
  • Preflight MUST be read-only and cheap enough for interactive use.
  • Writes MUST be chunked and resilient to partial failures.

UI Action Matrix (mandatory when Filament is changed)

Surface	Location	Header Actions	Inspect Affordance (List/Table)	Row Actions (max 2 visible)	Bulk Actions (grouped)	Empty-State CTA(s)	View Header Actions	Create/Edit Save+Cancel	Audit log?	Notes / Exemptions
Runbooks	/system/ops/runbooks	Preflight (read-only), Run… (write, confirm)	N/A	View run (after start)	None	None	N/A	N/A	Yes	Run… requires confirmation; typed confirm + reason required for All tenants.
Operation Runs	/system/ops/runs	N/A	List links to run detail (“View run”)	View	None	None	N/A	N/A	Yes	Run detail includes scope, actor, counts, outcome/status.

Key Entities (include if feature involves data)

• Runbook: A predefined operator action with preflight and run behavior.
• Operation Run: A tracked execution record storing scope, actor, status/outcome, and summary counts.
• Audit Event: Immutable security/ops log entries for preflight/run lifecycle.
• Operator Notification: A delivery record/target for failure alerts.
• Finding: Tenant-owned record whose lifecycle/workflow fields may be backfilled.

User Story 4 - Enterprise-grade UX polish for Ops surfaces (Priority: P2)

As a platform operator, the Ops surfaces should look and feel enterprise-grade with proper visual hierarchy, alert banners, structured card layouts, badge indicators, and metadata so I can quickly assess system state.

Why this priority: Operator trust and efficiency depend on clear, scannable UI. Raw text and flat layouts slow down triage.

Acceptance Scenarios:

1. Given the operator opens /system/ops/runbooks, Then the operator warning is rendered as a styled alert banner with icon (not plain text).
2. Given runbooks are listed, Then each runbook is rendered as a structured card with title, description, scope badge, and "Last run" metadata when available.
3. Given preflight results are displayed, Then stat values use consistent stat-card styling with labels and prominent values.
4. Given the operator opens /system/ops/runs/{id}, Then status and outcome are rendered as colored badges (consistent with the existing BadgeRenderer), and scope is shown as a badge/tag.
5. Given the run detail page, Then summary counts are rendered as a labeled grid (not only raw JSON).

Functional Requirements (UX Polish)

• FR-010 (Operator Warning Banner): The operator warning on /system/ops/runbooks MUST be rendered as a visually distinct alert banner with an exclamation-triangle icon, amber/warning coloring, and clear heading — matching project alert patterns.
• FR-011 (Runbook Card Layout): Each runbook MUST be rendered as a card with: title (semibold), description, scope badge (e.g., "All tenants"), and optional "Last run" timestamp + status badge when a previous run exists.
• FR-012 (Preflight Stat Cards): Preflight result values (affected, total scanned, estimated tenants) MUST be rendered in visually prominent stat cards with labeled headers.
• FR-013 (Run Detail Badges): Status and outcome on run detail pages MUST use the existing BadgeRenderer / BadgeCatalog system for colored badges with icons.
• FR-014 (Run Detail Summary Grid): Summary counts on run detail MUST be rendered as a labeled key-value grid, not a raw JSON dump (JSON viewer remains available as a disclosure fallback).

Success Criteria (mandatory)

Measurable Outcomes

• SC-001: In production-like environments, customers have zero UI affordances to trigger backfills/repairs in /admin.
• SC-002: A platform operator can start a runbook without SSH and reach "View run" in ≤ 3 user interactions from /system/ops/runbooks.
• SC-003: 100% of run attempts result in an operation run record and start/completion/failure audit events (with failure still recorded even if notifications fail).
• SC-004: Re-running the same runbook on the same scope after completion results in updated_count = 0 (idempotency).
• SC-005: Operator warning on runbooks page renders as a styled alert banner (not plain text).