TenantAtlas/specs/054-unify-runs-suitewide/spec.md

Feature Specification: Unified Operations Runs Suitewide (Except Restore Domain Model) (054)

Feature Branch: feat/054-unify-operations-runs-suitewide
Created: 2026-01-16
Status: Draft
Input: User description: "Eliminate run sprawl by adopting one canonical tenant-scoped operation run record for long-running actions across the product, surfaced consistently in Monitoring → Operations, while keeping restore as a separate domain workflow that is still visible via an adapter entry."

Clarifications

Session 2026-01-16

• Q: Welche Default-Retention soll 054 für canonical Operation Runs festlegen? → A: 90 days
• Q: Transition-Strategie in 054: schreiben wir canonical Runs parallel zu Legacy-Run-Tabellen, oder ersetzen wir sofort? → A: Parallel write (canonical + legacy)
• Q: For restore.execute, the spec mentions it acts as an "adapter entry" linking to the restore domain record. How should this be implemented? → A: Physical Row (Create a physical row in operation_runs that points to the restore record).
• Q: How should concurrency and deduplication (FR-009) be enforced at the database level? → A: Partial Unique Index (unique constraint on tenant_id, run_identity_hash where outcome is queued or running).
• Q: How should the initiator be modeled to support both users and system processes (FR-001)? → A: Nullable FK + Name Snapshot (user_id nullable FK + required initiator_name string).

User Scenarios & Testing (mandatory)

User Story 1 - See Every Supported Operation in Monitoring (Priority: P1)

As an operator, I want Monitoring → Operations to show all supported long-running operations for my tenant in one consistent list and detail view, so I can quickly answer what ran, who started it, whether it succeeded/partially succeeded/failed, and where to look next.

Why this priority: This is the core value: a single, tenant-scoped source of truth for operational visibility.

Independent Test: Trigger at least one run of each Phase 1 run producer, then verify each appears in Monitoring with consistent status/outcome semantics, safe failure summaries, and context links.

Acceptance Scenarios:

1. Given I am signed into tenant A, When I open Monitoring → Operations, Then I see only tenant A runs and can filter by run type, outcome bucket, time range, and initiator.
2. Given multiple run types exist, When I filter to inventory.sync, Then only inventory sync runs are shown.
3. Given a run exists, When I open its detail view, Then I can see initiator, run type, outcome bucket, timestamps, summary counts (if applicable), sanitized failures (if any), and links to relevant feature context/results.
4. Given restore execution exists, When I open Monitoring → Operations, Then I can see a restore.execute entry that links to the existing restore record (restore history remains owned by the restore domain record).
5. Given I am a Readonly user in tenant A, When I view Monitoring → Operations, Then I can view runs and details but I do not see any start/rerun/cancel/delete controls.
6. Given I attempt to access a run from another tenant (direct link or list), When I request it, Then access is denied and no run details are disclosed.

---

User Story 2 - Start Operations Without Blocking (Priority: P2)

As an operator, when I start a supported operation, I want immediate confirmation and a “View run” link so I can continue working while the operation runs in the background.

Why this priority: Removes long-running requests/timeouts and standardizes how operations are started and observed.

Independent Test: Start each Phase 1 operation from its owning UI and confirm the start returns quickly, includes “View run”, and the run progresses through queued/running into a terminal outcome.

Acceptance Scenarios:

1. Given I have permission to start a Phase 1 operation in tenant A, When I start it, Then I receive immediate confirmation with a “View run” link and the run is visible as queued or running.
2. Given I am a Readonly user in tenant A, When I attempt to start any Phase 1 operation, Then the system denies the request and does not create a new run.
3. Given the run reaches a terminal outcome, When that occurs, Then the initiating user receives an in-app notification including a short summary and a “View run” link.
4. Given background processing is unavailable, When I attempt to start an operation, Then I receive a clear message and the system MUST NOT claim it was queued.

---

User Story 3 - Duplicate Starts Reuse the Same Active Run (Priority: P3)

As an operator, I want accidental double-starts (double clicks, two admins, retries) to reuse the same active run so duplicate background work is avoided and results remain auditable.

Why this priority: Reduces load, prevents confusing duplicate outcomes, and makes operations safer under concurrency.

Independent Test: Start the same operation twice with identical effective inputs while the first is queued/running and verify the system reuses the active run.

Acceptance Scenarios:

1. Given an identical run is queued/running for a tenant, When another start request is made with the same effective inputs, Then the system reuses the existing run and does not start a second one.
2. Given two starts happen at nearly the same time, When the system resolves the race, Then at most one active run exists for that identity and both users are directed to it.

Edge Cases

• Background execution unavailable: start fails fast with a clear message; the system MUST NOT create misleading “queued” runs.
• Partial processing: at least one success and at least one failure yields “partially succeeded”, with per-item failures when applicable.
• Large run history: Monitoring remains usable with filters and defaults (recent runs, last 30 days).
• Permissions revoked mid-run: the run continues; visibility is evaluated at time of access.

Requirements (mandatory)

Constitution alignment (required): If this feature introduces any external tenant API calls or any write/change behavior, the spec MUST describe contract registry updates, safety gates (preview/confirmation/audit), tenant isolation, and tests.

Scope & Assumptions

Phase 1 adoption set (must be implemented):

• inventory.sync (Inventory “Sync now”)
• policy.sync (Policies “Sync now”)
• directory_groups.sync (Directory → Groups “Sync groups”)
• drift.generate (Drift “Generate drift now” / auto-on-open when eligible)
• backup_set.add_policies (Backup Sets “Add selected” / “Add policies”)

Restore visibility (adapter only):

• restore.execute appears as a canonical run entry that links to an existing restore domain record.
• Restore execution history remains owned by the restore domain record (not replaced in Phase 1).

Out of scope for 054 (explicit):

• Cross-tenant compare/promotion
• UI redesign/styling polish (separate UI polish work)
• Cancel/rerun/delete controls inside Monitoring hub (hub stays view-only)
• Replacing restore domain records with canonical runs
• A full settings UI for retention/notifications/etc.

Assumptions (defaults to remove ambiguity in Phase 1):

• Canonical run history retention defaults to 90 days, with no user-facing retention configuration in 054.
• System-initiated runs (if any) do not notify users by default in Phase 1.
• Transition strategy: write canonical runs in parallel with any existing legacy per-module run tables (where they exist); Monitoring uses canonical runs as the source of truth immediately.

Functional Requirements

• FR-001 Canonical Operation Run: System MUST represent each supported operation execution as a canonical, tenant-scoped operation run record that captures initiator (nullable user_id FK + initiator_name string), run type, lifecycle status/timestamps, outcome bucket, summary counts (when applicable), safe failure summaries, an idempotency identity for dedupe, and a safe context payload referencing “what this run was about”.
• FR-002 Run taxonomy: Run type MUST be stable and follow "<resource>.<action>".
• FR-003 Phase 1 run types: Phase 1 run types MUST include inventory.sync, policy.sync, directory_groups.sync, drift.generate, backup_set.add_policies, plus restore.execute implemented as a physical operation_runs record (adapter) pointing to the domain entity.
• FR-004 Monitoring lists all canonical runs: Monitoring → Operations MUST list canonical runs for the active tenant with filters for run type, outcome bucket, time range, and initiator; default sort is most recent first; default time window is last 30 days.
• FR-005 Run detail: Run detail MUST show initiator, run type, outcome bucket, timestamps (created/started/finished), summary counts (when applicable), sanitized failures (including per-item failures when applicable), and contextual links to owning feature surfaces/results.
• FR-006 View-only hub: Monitoring hub MUST be view-only (no start/rerun/cancel/delete controls) and MUST link back to owning feature surfaces.
• FR-007 Start surfaces always enqueue: Every Phase 1 start surface MUST authorize start, create/reuse a canonical run (dedupe), dispatch background execution, and return immediately with confirmation + “View run”.
• FR-008 No remote work in interactive request: Start surfaces MUST NOT perform remote work inline; long-running work happens in background execution.
• FR-009 Deterministic idempotency: For each run type, the system MUST define a deterministic identity for “identical run” based on tenant + effective inputs; initiator MUST NOT be part of identity. Enforcement: Uniqueness MUST be enforced via a partial unique index on (tenant_id, run_identity_hash) where outcome is queued or running.
• FR-010 Phase 1 identity rules: Identity rules MUST be defined at least as follows:
  • inventory.sync: tenant + selection scope
  • policy.sync: tenant + effective policy scope
  • directory_groups.sync: tenant + selection (Phase 1 default: “all groups”)
  • backup_set.add_policies: tenant + backup set + selected policies + option flags (if exposed)
  • drift.generate: tenant + scope key + baseline/current comparison inputs
• FR-011 Outcome buckets: Monitoring MUST present consistent outcome buckets: queued, running, succeeded, partially succeeded, failed.
• FR-012 Partial vs failed: “Partially succeeded” means at least one success and at least one failure; “Failed” means zero successes or cannot proceed.
• FR-013 Failure details are safe + useful: Failures MUST be persisted and displayed as stable reason codes and short sanitized messages; failures MUST NOT include secrets/tokens/credentials/PII or full external payload dumps.
• FR-014 Related links: Run detail MUST include contextual links where applicable (e.g., drift findings, backup set, inventory results, directory groups, restore detail for restore.execute).
• FR-015 Notifications: System MUST emit in-app notifications for “queued” (after start) and terminal outcomes for Phase 1 runs; notifications MUST include a short summary and a “View run” link; recipients are the initiating user only.
• FR-016 Tenant isolation: All run list/detail access MUST be tenant-scoped; cross-tenant access MUST be denied without disclosing run details.
• FR-017 No render-time remote calls: Monitoring pages MUST be render-safe and MUST NOT depend on external service calls during render.
• FR-018 Roles & permissions: Roles Owner, Manager, Operator, and Readonly MUST be able to view runs; only Owner, Manager, Operator may start operations; Readonly is strictly view-only.

Key Entities (include if feature involves data)

• Canonical Operation Run: A tenant-scoped record representing the lifecycle of a long-running operation, including run type, initiator (nullable user_id FK + initiator_name string), lifecycle state/timestamps, outcome bucket, summary counts, safe failure summaries, idempotency identity (uniqueness enforced by DB index on active runs), and safe context references.
• Restore domain record (exception): Restore remains a domain workflow record with richer semantics and history. Monitoring shows restore activity through a physical operation_runs row (adapter) that links back to the restore record, without replacing it.

Success Criteria (mandatory)

Measurable Outcomes

• SC-001: Operators can answer “what ran, when, and did it succeed?” for any Phase 1 run in under 1 minute using Monitoring → Operations.
• SC-002: Starting a Phase 1 operation returns confirmation + “View run” link within 2 seconds under normal conditions.
• SC-003: Duplicate starts reuse the same active run in at least 99% of attempts under normal conditions.
• SC-004: No secrets/tokens/credentials/PII appear in persisted failures or notifications (verified by tests).