TenantAtlas/specs/049-backup-restore-job-orchestration/spec.md

Feature Specification: Backup/Restore Job Orchestration (049)

Feature Branch: feat/049-backup-restore-job-orchestration
Created: 2026-01-11
Status: Draft
Input: Ensure Backup/Restore “start/execute” actions never run inline in an interactive request; they run via background processing with run records and visible progress.

Purpose

All Backup/Restore “Start/Execute” actions run exclusively via background processing with Run Records and visible progress. This prevents timeouts, double-click duplication, throttling issues, and improves reliability at MSP scale.

Non-Goals (Phase 1)

• No new directory/group inventory or name resolution features (separate initiative)
• No changes to external service contracts unless required for orchestration safety
• No new promotion feature (e.g., DEV→PROD) (separate initiative)

Clarifications

Session 2026-01-11

• Q: For FR-004 Idempotency, what should happen when the admin starts the same operation again for the same tenant + target while one is still queued/running? → A: Reuse existing run if identical is queued/running; allow a new run only after terminal.
• Q: For FR-002 status lifecycle, do we support canceling runs in Phase 1? → A: No cancel in Phase 1.
• Q: For FR-003 Progress visibility, which UI surfaces are required in Phase 1? → A: Phase 1 requires Run detail progress (counts/status) + DB notifications; Phase 2 adds a required global progress widget for all run types.
• Q: For FR-004, how should we define the “target object” used for de-duplication? → A: Dedupe key uses (tenant + operation type + target object id).
• Q: For FR-005 per-item outcome persistence, what is the item granularity for counts + item results in Phase 1? → A: Per internal DB record (e.g., restore/backup item rows).

User Scenarios & Testing (mandatory)

User Story 1 - Capture snapshot runs in background (Priority: P1)

An admin can start a “capture snapshot” operation without the UI hanging or timing out, and can see progress plus the final result.

Why this priority: Snapshot capture is a core workflow and a common source of long-running requests.

Independent Test: Starting a snapshot capture immediately returns to the UI with a queued Run Record that later transitions to a terminal state (success/failed/partial) and can be inspected.

Acceptance Scenarios:

1. Given an admin has access to a tenant, When they start “capture snapshot”, Then the UI confirms it was queued and shows a link to the Run Record.
2. Given a capture snapshot run is executing, When the admin views the run, Then they see progress (items done vs total) and any safe error summaries.

---

User Story 2 - Backup set create/capture runs in background (Priority: P2)

An admin can create a backup set and optionally start a capture/sync operation without the request doing heavy work.

Why this priority: Creating backup sets is frequent and should not be coupled to long-running capture logic.

Independent Test: Creating a backup set returns quickly and any capture/sync work appears as a run with progress.

Acceptance Scenarios:

1. Given an admin creates a backup set with capture enabled, When they submit, Then the backup set is created and a capture run is queued.

---

User Story 3 - Restore runs in background with per-item results (Priority: P1)

An admin can start a “restore to Intune” or “re-run restore” operation as a background run and later inspect item-level outcomes and errors.

Why this priority: Restore is high-impact and must be resilient, observable, and safe under retries.

Independent Test: Starting restore creates a Run Record and item results that remain accessible even if the external service is unavailable.

Acceptance Scenarios:

1. Given an admin starts a restore, When they confirm the action, Then the UI queues a run and returns immediately (no long-running request).
2. Given a restore run finishes with mixed outcomes, When the admin views the run details, Then they see succeeded/failed counts and a safe error summary per failed item.
3. Given an admin executes a live restore, When the run is queued/executed, Then an auditable event is recorded that links to the run.

---

User Story 4 - Dry-run/preview runs in background (Priority: P2)

An admin can run a dry-run/preview without UI timeouts, and the preview results are persisted and shown in the UI.

Why this priority: Preview supports safe change management and must remain usable even when the external service is slow or down.

Independent Test: Starting preview immediately creates a run; once finished, preview outputs are visible and reusable.

Acceptance Scenarios:

1. Given an admin starts a preview run, When the run completes, Then the UI shows preview results without requiring re-execution.
2. Given an admin starts a preview/dry-run, When the run executes, Then no write/change is performed against the external system.

Edge Cases

• Double-clicking an action rapidly
• Retrying while an identical run is already queued or running
• External service is unavailable (e.g., throttling or outage)
• A run gets stuck or exceeds expected duration
• Permissions change after a run was queued

Requirements (mandatory)

Constitution alignment (required): If this feature introduces any Microsoft Graph calls or any write/change behavior, the spec MUST describe contract registry updates, safety gates (preview/confirmation/audit), tenant isolation, and tests.

Functional Requirements

• FR-001 Job-only execution: The system MUST execute the following operations via background processing and MUST NOT perform heavy work inline during the interactive request:

  • Capture snapshot
  • Backup set create with capture/sync (when capture is triggered)
  • Restore to Intune
  • Re-run restore
  • Restore dry-run/preview

• FR-002 Run Records: Each operation start MUST create (or deterministically re-use) a Run Record before the work begins, containing:

  • Tenant identity
  • Initiator identity (user reference or audit reference)
  • Operation type and optional target object reference
  • Status lifecycle: queued → running → (succeeded | failed | partial)
  • Started/finished timestamps
  • Item counts: total / succeeded / failed
  • Safe error code and safe error context (no secrets)

• FR-003 Progress visibility: While a run is executing, the system MUST:

  • Emit in-app notifications for key state transitions (queued/running/completed/failed)
  • Provide a Run detail view that shows progress (status + item counts)

  Phase 1 does not require a global progress widget. Phase 2 MUST add a global progress widget, required for all run types.

• FR-004 Idempotency & concurrency control: The system MUST prevent uncontrolled duplicate execution due to double-clicks/retries by enforcing a deterministic de-duplication rule keyed by (tenant + operation type + target object) or (tenant + run id). When an identical run is already queued/running, the UI MUST show “already queued/running” and link to the existing run.

  Clarification: For an identical start attempt while a run is queued or running, the system MUST re-use the existing Run Record and MUST NOT create a new run. A new run MAY be started only after the existing run reaches a terminal state.

  Clarification: For Phase 1, the default de-duplication key is (tenant + operation type + target object id).

• FR-005 Deterministic outcome persistence: The system MUST persist per-item outcomes for operations that act on multiple items, including status and a safe error summary, so results can be viewed later without relying on logs.

  Clarification: In Phase 1, “item” refers to the internal DB record being acted on (e.g., a restore/backup item row). Counts (total/succeeded/failed) MUST be derived from these persisted item results.

• FR-006 Tenant isolation & authorization: Run visibility and execution MUST be tenant-scoped. Only authorized admins can start operations, and users MUST NOT be able to view or start runs across tenants.

• FR-007 Safety rules: Preview/dry-run MUST be safe (no writes). Live restore MUST remain guarded with explicit confirmation and an auditable trail consistent with existing safety practices.

• FR-008 Resilience (Post-MVP / Phase 2): The system MUST handle external service throttling/outages gracefully, including retries with backoff when appropriate, and MUST end runs in a clear terminal state (failed/partial) rather than silently failing.

  Note: MVP/Phase 1 relies on existing retry behavior where present; standardized backoff + jitter hardening is scheduled post-MVP.

• FR-009 Safe logging & data minimization: The system MUST NOT store secrets/tokens in Run Records, notifications, or error contexts. Error context MUST be limited to a defined, safe set of fields.

Acceptance Checks

• Starting any in-scope operation returns quickly with a queued Run Record link.
• A Run Record always exists before background work begins and reaches a terminal state.
• Phase 1 does not support canceling runs.
• Progress and state changes are visible via Run detail view and in-app notifications.
• Phase 2 adds a global progress widget for all run types.
• Duplicate start attempts for the same tenant + operation + target do not create uncontrolled duplicate execution.
• Duplicate start attempts for the same tenant + operation + target while a run is queued/running re-use the existing run and link to it.
• Item-level outcomes and safe error summaries are viewable after completion.
• Run counts reflect persisted internal item results.
• Preview/dry-run never performs writes.
• Unauthorized users cannot start runs for a tenant they do not belong to.
• Users cannot list/view run records across tenants.
• Live restore creates an auditable event linked to the run.

Key Entities (include if feature involves data)

• Run Record: A tenant-scoped record representing one started operation and its lifecycle, progress, and summary outcome.
• Run Item Result: A tenant-scoped record representing the outcome for a single item processed as part of a Run Record.
• Notification Event: A tenant-scoped event surfaced to the admin UI to communicate run state changes.

Success Criteria (mandatory)

Measurable Outcomes

• SC-001: For 95% of operation starts, the UI confirms “queued” within 2 seconds.
• SC-002: Double-clicking an operation start results in at most one queued/running run for the same tenant + operation + target.
• SC-003: 99% of runs end in a clear terminal state (succeeded/failed/partial) with a human-readable summary.
• SC-004: Admins can locate the latest run status for an operation in under 30 seconds without requiring access to system logs.

Note: “canceled” is reserved for Phase 2+ (Phase 1 has no cancel support).

Assumptions

• This feature builds on the UI safety constraints from 048: admin pages must remain usable even when the external service API is unavailable.
• Run Records and item results are retained long enough to support operational troubleshooting and audits, with retention managed as a separate policy.