TenantAtlas/specs/054-unify-runs-suitewide/spec.md

# Feature Specification: Unified Operations Runs Suitewide (Except Restore Domain Model) (054)

**Feature Branch**: `feat/054-unify-operations-runs-suitewide`  
**Created**: 2026-01-16  
**Status**: Draft  
**Input**: User description: "Eliminate run sprawl by adopting one canonical tenant-scoped operation run record for long-running actions across the product, surfaced consistently in Monitoring → Operations, while keeping restore as a separate domain workflow that is still visible via an adapter entry."

## Clarifications

### Session 2026-01-16

- Q: Welche Default-Retention soll 054 für canonical Operation Runs festlegen? → A: 90 days
- Q: Transition-Strategie in 054: schreiben wir canonical Runs parallel zu Legacy-Run-Tabellen, oder ersetzen wir sofort? → A: Parallel write (canonical + legacy)
- Q: For `restore.execute`, the spec mentions it acts as an "adapter entry" linking to the restore domain record. How should this be implemented? → A: Physical Row (Create a physical row in `operation_runs` that points to the restore record).
- Q: How should concurrency and deduplication (FR-009) be enforced at the database level? → A: Partial Unique Index (unique constraint on `tenant_id, run_identity_hash` where status is `queued` or `running`).
- Q: How should the `initiator` be modeled to support both users and system processes (FR-001)? → A: Nullable FK + Name Snapshot (`user_id` nullable FK + required `initiator_name` string).

### Session 2026-01-17

- Q: Sollen `backup_schedule.run_now` und `backup_schedule.retry` in 054 zur Phase-1-Adoption (must be implemented) gehören? → A: Yes — both are Phase 1 in 054 (OperationRun producers + worker tracking).
- Q: Wenn Queue-Dispatch fehlschlägt (Background Processing unavailable), sollen wir trotzdem einen `OperationRun` anlegen und ihn sofort als fehlgeschlagen abschließen? → A: Yes — create an `OperationRun` and immediately complete it as `failed` (e.g., failure code `queue.dispatch_failed`); show a clear error and MAY include a “View run” link.
- Q: Wenn ein Start deduped wird (Run wird wiederverwendet), wer soll die In‑App Notifications (“queued” + terminal outcome) bekommen? → A: Only the original initiator (`operation_runs.user_id`); no additional notifications are sent to the second starter on reuse.
- Q: Für `restore.execute`: In welchen `RestoreRunStatus`-Phasen soll überhaupt ein `OperationRun`-Adapter‑Row erzeugt/angezeigt werden? → A: From `previewed` onwards (previewed + execution statuses); no adapter row for `draft`/`scoped`/`checked`.
- Q: Wenn der `restore.execute` Adapter bereits ab `RestoreRunStatus=previewed` sichtbar ist: welchen `OperationRun`-State sollen wir für diese Phase setzen? → A: `status=queued`, `outcome=pending` (until `running`, then `completed` + terminal outcome).
- Q: RBAC Wizard (`TenantResource`) – wie funktioniert Group Search? → A: Group search is delegated-Graph-based and the picker MUST be disabled without delegated auth.
- Q: Restore Wizard (`RestoreRunResource`) – Group Mapping Phase: Graph oder DB-only? → A: DB-only via Directory Cache (`entra_groups`), no Graph calls during mapping; helper text is always shown (fallback included).

## User Scenarios & Testing *(mandatory)*

### User Story 1 - See Every Supported Operation in Monitoring (Priority: P1)

As an operator, I want Monitoring → Operations to show all supported long-running operations for my tenant in one consistent list and detail view, so I can quickly answer what ran, who started it, whether it succeeded/partially succeeded/failed, and where to look next.

**Why this priority**: This is the core value: a single, tenant-scoped source of truth for operational visibility.

**Independent Test**: Trigger at least one run of each Phase 1 run producer, then verify each appears in Monitoring with consistent status/outcome semantics, safe failure summaries, and context links.

**Acceptance Scenarios**:

1. **Given** I am signed into tenant A, **When** I open Monitoring → Operations, **Then** I see only tenant A runs and can filter by run type, run state (queued/running/terminal outcome), time range, and initiator.
2. **Given** multiple run types exist, **When** I filter to `inventory.sync`, **Then** only inventory sync runs are shown.
3. **Given** a run exists, **When** I open its detail view, **Then** I can see initiator, run type, run state (queued/running/terminal outcome), timestamps, summary counts (if applicable), sanitized failures (if any), and links to relevant feature context/results.
4. **Given** a restore run has reached `previewed` or later, **When** I open Monitoring → Operations, **Then** I can see a `restore.execute` entry that links to the existing restore record (restore history remains owned by the restore domain record).
5. **Given** I am a `Readonly` user in tenant A, **When** I view Monitoring → Operations, **Then** I can view runs and details but I do not see any start/rerun/cancel/delete controls.
6. **Given** I attempt to access a run from another tenant (direct link or list), **When** I request it, **Then** access is denied and no run details are disclosed.

---

### User Story 2 - Start Operations Without Blocking (Priority: P2)

As an operator, when I start a supported operation, I want immediate confirmation and a “View run” link so I can continue working while the operation runs in the background.

**Why this priority**: Removes long-running requests/timeouts and standardizes how operations are started and observed.

**Independent Test**: Start each Phase 1 operation from its owning UI and confirm the start returns quickly, includes “View run”, and the run progresses through queued/running into a terminal outcome.

**Acceptance Scenarios**:

1. **Given** I have permission to start a Phase 1 operation in tenant A, **When** I start it, **Then** I receive immediate confirmation with a “View run” link and the run is visible as queued or running.
2. **Given** I am a `Readonly` user in tenant A, **When** I attempt to start any Phase 1 operation, **Then** the system denies the request and does not create a new run.
3. **Given** the run reaches a terminal outcome, **When** that occurs, **Then** the initiating user receives an in-app notification including a short summary and a “View run” link.
4. **Given** background processing is unavailable, **When** I attempt to start an operation, **Then** I receive a clear message and the system MUST NOT claim it was queued.
   - If an `OperationRun` record was created during the attempt, it MUST be completed immediately with outcome `failed` (never left `queued`) and MAY be linked via “View run”.

---

### User Story 3 - Duplicate Starts Reuse the Same Active Run (Priority: P3)

As an operator, I want accidental double-starts (double clicks, two admins, retries) to reuse the same active run so duplicate background work is avoided and results remain auditable.

**Why this priority**: Reduces load, prevents confusing duplicate outcomes, and makes operations safer under concurrency.

**Independent Test**: Start the same operation twice with identical effective inputs while the first is queued/running and verify the system reuses the active run.

**Acceptance Scenarios**:

1. **Given** an identical run is queued/running for a tenant, **When** another start request is made with the same effective inputs, **Then** the system reuses the existing run and does not start a second one.
2. **Given** two starts happen at nearly the same time, **When** the system resolves the race, **Then** at most one active run exists for that identity and both users are directed to it.

### Edge Cases

- Background execution unavailable: start fails fast with a clear message; if an `OperationRun` record was created, it MUST be immediately completed as `failed` (e.g., `queue.dispatch_failed`) and MUST NOT be left `queued`.
- Partial processing: at least one success and at least one failure yields “partially succeeded”, with per-item failures when applicable.
- Large run history: Monitoring remains usable with filters and defaults (recent runs, last 30 days).
- Permissions revoked mid-run: the run continues; visibility is evaluated at time of access.

## Requirements *(mandatory)*

**Constitution alignment (required):** If this feature introduces any external tenant API calls or any write/change behavior,
the spec MUST describe contract registry updates, safety gates (preview/confirmation/audit), tenant isolation, and tests.

### Scope & Assumptions

**Phase 1 adoption set (must be implemented):**

- `inventory.sync` (Inventory “Sync now”)
- `policy.sync` (Policies “Sync now”)
- `directory_groups.sync` (Directory → Groups “Sync groups”)
- `drift.generate` (Drift “Generate drift now” / auto-on-open when eligible)
- `backup_set.add_policies` (Backup Sets “Add selected” / “Add policies”)
- `backup_schedule.run_now` (Backup Schedules “Run now”)
- `backup_schedule.retry` (Backup Schedules “Retry”)

**Restore visibility (adapter only):**

- `restore.execute` appears as a canonical run entry that links to an existing restore domain record.
- The adapter row MUST be created/visible only once a restore run reaches `previewed` (or later) and MUST NOT be created for `draft`, `scoped`, or `checked`.
- When the restore run is `previewed`, the adapter `OperationRun` MUST use `status=queued` and `outcome=pending`.
- Restore execution history remains owned by the restore domain record (not replaced in Phase 1).

**Out of scope for 054 (explicit):**

- Cross-tenant compare/promotion
- UI redesign/styling polish (separate UI polish work)
- Cancel/rerun/delete controls inside Monitoring hub (hub stays view-only)
- Replacing restore domain records with canonical runs
- A full settings UI for retention/notifications/etc.
- Implementing or validating `AuditLog` behavior for audit-only actions (FR-019) beyond actions explicitly changed by 054

**Assumptions (defaults to remove ambiguity in Phase 1):**

- Canonical run history retention defaults to 90 days, with no user-facing retention configuration in 054.
- System-initiated runs (if any) do not notify users by default in Phase 1.
- Transition strategy: write canonical runs in parallel with any existing legacy per-module run tables (where they exist); Monitoring uses canonical runs as the source of truth immediately.

**Run vs Audit-only Adoption Matrix (Phase 1):**

| Feature Area | Action | Tracking | run_type / audit action |
|-------------|--------|----------|--------------------------|
| Policies | Sync now | OperationRun | `policy.sync` |
| Policies | Ignore policy | Audit-only | `policy.ignore` |
| Policies | Export to backup | OperationRun (queued) | `policy.export_backup` |
| Policy Versions | Capture snapshot | OperationRun | `policy.capture_snapshot` |
| Policy Versions | Prune versions | Audit-only | `policy_versions.prune` |
| Policy Versions | Archive versions | Audit-only | `policy_versions.archive` |
| Inventory | Sync now | OperationRun | `inventory.sync` |
| Directory Groups | Sync groups | OperationRun | `directory_groups.sync` |
| Drift | Generate drift | OperationRun | `drift.generate` |
| Backup Sets | Add policies | OperationRun | `backup_set.add_policies` |
| Backup Sets | Archive | Audit-only (DB-only) | `backup_set.archive` |
| Backup Sets | Restore (bulk) | OperationRun | `backup_set.restore` |
| Backup Sets | Force delete | Audit-only (admin-only) | `backup_set.force_delete` |
| Backup Schedules | Run now | OperationRun | `backup_schedule.run_now` |
| Backup Schedules | Retry | OperationRun | `backup_schedule.retry` |
| Backup Schedules | Edit | Audit-only | `backup_schedule.edit` |
| Backup Schedules | Delete | Audit-only | `backup_schedule.delete` |
| Tenants | Sync tenant | OperationRun | `tenant.sync` |
| Tenants | Admin consent | Audit-only | `tenant.admin_consent` |
| Tenants | Verify configuration | Audit-only | `tenant.verify_config` |
| Tenants | Setup Intune RBAC | Audit-only | `tenant.setup_rbac` |
| Tenants | Deactivate | Audit-only | `tenant.deactivate` |
| Restore | Execute restore | OperationRun (adapter) | `restore.execute` (context → `restore_run_id`) |

**Rule**: If an action is queued/background, long-running, or requires remote/external calls (e.g., Microsoft Graph),
it MUST be tracked as an OperationRun. Only fast DB-only changes MAY be Audit-only.

### Functional Requirements

- **FR-001 Canonical Operation Run**: System MUST represent each supported operation execution as a canonical, tenant-scoped operation run record that captures initiator (nullable `user_id` FK + `initiator_name` string), run type, lifecycle status/timestamps, terminal outcome (pending while active), summary counts (when applicable), safe failure summaries, an idempotency identity for dedupe, and a safe context payload referencing “what this run was about”.
  - **Status semantics**: `status` represents lifecycle stage (`queued` → `running` → `completed`).
  - **Outcome semantics (stored tokens)**: `outcome` stores machine tokens: `pending` while active, otherwise `succeeded` / `partially_succeeded` / `failed`.
    - **UI labels**: Monitoring displays human labels derived from stored tokens (e.g., `partially_succeeded` → “Partially succeeded”).
    - **Reserved**: `cancelled` is reserved for future use and MUST NOT be produced by 054 (Monitoring hub has no cancel controls).
  - **Context safety**: `context` MUST be sanitized and MUST include only safe references (e.g., stable IDs, selection scope keys, correlation IDs). It MUST NOT include secrets/tokens/credentials, personal data, or full external payload dumps.
- **FR-002 Run taxonomy**: Run type MUST be stable and follow `"<resource>.<action>"`.
- **FR-003 Phase 1 run types**: Phase 1 run types MUST include `inventory.sync`, `policy.sync`, `directory_groups.sync`, `drift.generate`, `backup_set.add_policies`, `backup_schedule.run_now`, `backup_schedule.retry`, plus `restore.execute` implemented as a physical `operation_runs` record (adapter) pointing to the domain entity.
- **FR-004 Monitoring lists all canonical runs**: Monitoring → Operations MUST list canonical runs for the active tenant with filters for run type, run state (queued/running/terminal outcome), time range, and initiator; default sort is most recent first; default time window is last 30 days.
- **FR-005 Run detail**: Run detail MUST show initiator, run type, run state (queued/running/terminal outcome), timestamps (created/started/finished), summary counts (when applicable), sanitized failures (including per-item failures when applicable), and contextual links to owning feature surfaces/results.
- **FR-006 View-only hub**: Monitoring hub MUST be view-only (no start/rerun/cancel/delete controls) and MUST link back to owning feature surfaces.
- **FR-007 Start surfaces always enqueue**: Every Phase 1 start surface MUST authorize start, create/reuse a canonical run (dedupe), dispatch background execution, and return immediately with confirmation + “View run”.
- **FR-008 No remote work in interactive request**: Start surfaces MUST NOT perform remote work inline; long-running work happens in background execution.
- **FR-009 Deterministic idempotency**: For each run type, the system MUST define a deterministic identity for “identical run” based on tenant + effective inputs; initiator MUST NOT be part of identity. **Enforcement**: Uniqueness MUST be enforced via a partial unique index on `(tenant_id, run_identity_hash)` where status is `queued` or `running`.
- **FR-010 Phase 1 identity rules**: Identity rules MUST be defined at least as follows:
  - `inventory.sync`: tenant + selection scope
  - `policy.sync`: tenant + effective policy scope
  - `directory_groups.sync`: tenant + selection (Phase 1 default: “all groups”)
  - `backup_set.add_policies`: tenant + backup set + selected policies + option flags (if exposed)
  - `backup_schedule.run_now`: tenant + backup schedule id
  - `backup_schedule.retry`: tenant + backup schedule id
  - `drift.generate`: tenant + scope key + baseline/current comparison inputs
- **FR-011 Run state presentation**: Monitoring MUST present a consistent run state using a single display bucket derived from lifecycle status and terminal outcome:
  - If status is `queued` or `running`, display that status.
  - If status is `completed`, display the terminal outcome derived from the stored token (`succeeded`, `partially_succeeded`, or `failed`) using the UI label mapping.
- **FR-012 Partial vs failed (terminal outcomes)**: “Partially succeeded” (`partially_succeeded`) means at least one success and at least one failure; “Failed” (`failed`) means zero successes or cannot proceed.
- **FR-013 Failure details are safe + useful**: Failures MUST be persisted and displayed as stable reason codes and short sanitized messages; failures MUST NOT include secrets/tokens/credentials/PII or full external payload dumps.
  - **Reason codes** MUST be stable, machine-readable identifiers (lowercase, dot-separated), e.g. `graph.throttled`, `auth.forbidden`, `validation.invalid_input`, `unexpected.exception`.
  - **Messages** MUST be short (≤ 200 characters), sanitized, and written for operators (no secrets/tokens/credentials/PII; no raw external payloads). If needed, messages MAY include a non-sensitive correlation identifier.
- **FR-014 Related links**: Run detail MUST include contextual links where applicable (e.g., drift findings, backup set, inventory results, directory groups, restore detail for `restore.execute`).
- **FR-015 Notifications**: System MUST emit in-app notifications for “queued” (after start) and terminal outcomes for Phase 1 runs; notifications MUST include a short summary and a “View run” link; recipients are the initiating user only.
  - If a start request reuses an existing active run (dedupe), the run initiator (as stored on the `OperationRun`) remains the sole notification recipient; the second starter receives no additional notifications.
- **FR-016 Tenant isolation**: All run list/detail access MUST be tenant-scoped; cross-tenant access MUST be denied without disclosing run details.
- **FR-017 No render-time remote calls**: Monitoring pages MUST be render-safe and MUST NOT depend on external service calls during render.
- **FR-018 Roles & permissions**: Roles `Owner`, `Manager`, `Operator`, and `Readonly` MUST be able to view runs; only `Owner`, `Manager`, `Operator` may start operations; `Readonly` is strictly view-only.
- **FR-019 Audit-only actions (no OperationRun)**: Actions that are DB-only and complete within ≤2 seconds under normal
  conditions MAY be executed without an OperationRun, as long as they do not start long-running background execution and
  do not require any remote/external calls.
  - **054 scope note**: 054 does not implement or modify audit-only actions. If any audit-only action is touched as part
    of implementing 054 in the future, it MUST comply with this requirement and MUST be covered by tests.
  If such an action is security-relevant or changes operational behavior (e.g., “Ignore policy”, “Deactivate tenant”,
  “Admin consent”, “Prune versions”, “Force delete”), it MUST write exactly one tenant-scoped AuditLog entry with, at minimum:
  - `tenant_id`
  - `actor_user_id`
  - `action` (stable action identifier, e.g., `policy.ignore`)
  - `target_type`, `target_id`
  - `before` / `after` (sanitized JSON) **or** `diff` (sanitized JSON)
  - `created_at`
  **Trigger guidance (to make classification reviewable)**:
  - “Security-relevant” includes actions that grant/revoke access, change authorization posture, change admin consent, or otherwise modify who/what can read/write tenant data.
  - “Operational behavior change” includes actions that change what the system will do in future runs (e.g., ignore/exclude resources, enable/disable schedules, retention/prune/archive actions, force deletes).
  - If unclear whether an Audit-only action is security/ops-relevant, the default is to treat it as such and write an AuditLog entry.
  **Sanitization (AuditLog before/after/diff)**:
  - AuditLog payloads MUST include only the minimum fields needed to understand the change.
  - AuditLog payloads MUST NOT include secrets/tokens/credentials, personal data, or full external payload dumps.
  - If a field is sensitive, it MUST be omitted or replaced with a non-sensitive placeholder (e.g., `"[REDACTED]"`).
  Monitoring/Operations remains reserved for OperationRun-tracked long-running/queued operations.
  **Acceptance checks (testable)**:
  - Audit-only action creates no OperationRun.
  - Audit-only action creates exactly one AuditLog event containing the required fields.
  - Audit-only action is tenant-scoped; cross-tenant access is forbidden and MUST NOT create AuditLog entries.

### Key Entities *(include if feature involves data)*

- **Canonical Operation Run**: A tenant-scoped record representing the lifecycle of a long-running operation, including run type, initiator (nullable `user_id` FK + `initiator_name` string), lifecycle state/timestamps, terminal outcome, summary counts, safe failure summaries, idempotency identity (uniqueness enforced by DB index on active runs), and safe context references.
- **Restore domain record (exception)**: Restore remains a domain workflow record with richer semantics and history. Monitoring shows restore activity through a physical `operation_runs` row (adapter) that links back to the restore record, without replacing it.

## Success Criteria *(mandatory)*

### Measurable Outcomes

- **SC-001**: Operators can answer “what ran, when, and did it succeed?” for any Phase 1 run in under 1 minute using Monitoring → Operations.
- **SC-002**: Starting a Phase 1 operation returns confirmation + “View run” link within 2 seconds under normal conditions.
- **SC-003**: Duplicate starts reuse the same active run in at least 99% of attempts under normal conditions.
- **SC-003 Measurement scope (definition)**: An “attempt” counts when a start request is made for an operation with identical effective inputs while an identical run is already `queued` or `running`. The success condition is that the system reuses the existing active run reference rather than creating a second active run. “Normal conditions” exclude infrastructure outages (e.g., database unavailable) that prevent either run creation or dedupe evaluation.
- **SC-004**: No secrets/tokens/credentials/PII appear in persisted failures or notifications (verified by tests).