TenantAtlas/specs/054-unify-runs-suitewide/research.md

# Research: Unified Operations Runs Suitewide

## 1. Technical Context & Unknowns

**Unknowns Resolved**:
- **Transition Strategy**: Parallel write. We will maintain existing legacy tables (e.g., `inventory_sync_runs`, `restore_runs`) for now but strictly use `operation_runs` for the Monitoring UI.
- **Restore Adapter**: `RestoreRun` remains the domain source of truth. An `OperationRun` adapter row will be created once a restore run reaches `previewed` (and later statuses), and will be kept in sync via `RestoreRun` lifecycle events or service-layer wrapping.
- **Run Logic Location**: Existing jobs like `RunInventorySyncJob` will be updated to manage the `OperationRun` state.
- **Concurrency**: Enforced by partial unique index on `(tenant_id, run_identity_hash)` where status is active (`queued`, `running`).
- **Dispatch Failure Semantics**: If queue dispatch fails, the system will immediately complete the run as `failed` (e.g., `queue.dispatch_failed`) and show a clear UI message (never leaving misleading queued runs).
- **Notifications on Dedupe**: Only the original initiator (`operation_runs.user_id`) receives queued/terminal notifications; reusers of an active run do not get additional notifications.

## 2. Technology Choices

| Area | Decision | Rationale | Alternatives |
|------|----------|-----------|--------------|
| **Schema** | `operation_runs` table | Centralized table allows simple, performant Monitoring queries without complex UNIONs across disparate legacy tables. | Virtual UNION view (Complex, harder to paginate/sort efficiently). |
| **Restore Integration** | Physical Adapter Row | Decouples Monitoring from Restore domain specifics. Allows uniform "list all runs" queries. The `context` JSON column will store `{ "restore_run_id": ... }`. | Polymorphic relation (Overhead for a single exception). |
| **Idempotency** | DB Partial Unique Index | Hard guarantee against race conditions. Simpler than distributed locks (Redis) which can expire or fail. | Redis Lock (Soft guarantee), Application check (Race prone). |
| **Initiator** | Nullable FK + Name | Handles both Users (FK) and System/Scheduler (Name "System") uniformly. | Polymorphic relation (Overkill for simple auditing). |

## 3. Implementation Patterns

### Canonical Run Lifecycle
1.  **Start Request**:
    -   Compute `run_identity_hash` from inputs.
    -   Attempt `INSERT` into `operation_runs` (idempotent; enforced by partial unique index for active runs).
    -   If an active run exists, return it (Idempotency).
    -   If new, dispatch the background Job.
    -   If dispatch fails, immediately mark the run `status=completed`, `outcome=failed` with a safe failure code such as `queue.dispatch_failed`.
2.  **Job Execution**:
    -   Update status to `running`.
    -   Perform work.
    -   Update status to `completed` and set terminal outcome (`succeeded` / `partially_succeeded` / `failed` / `cancelled`).
3.  **Restore Adapter**:
    -   Create the adapter row only once `RestoreRunStatus=previewed` (or later) is reached.
    -   Map `RestoreRunStatus=previewed` to `OperationRun.status=queued` and `OperationRun.outcome=pending`.
    -   Keep the adapter updated as the restore progresses:
        - `queued` → `status=queued`, `outcome=pending`
        - `running` → `status=running`, `outcome=pending`
        - `completed` → `status=completed`, `outcome=succeeded`
        - `partial` → `status=completed`, `outcome=partially_succeeded`
        - `failed` → `status=completed`, `outcome=failed`
        - `cancelled` → `status=completed`, `outcome=cancelled`

### Data Model
```sql
CREATE TABLE operation_runs (
    id BIGSERIAL PRIMARY KEY,
    tenant_id BIGINT NOT NULL REFERENCES tenants(id),
    user_id BIGINT NULL REFERENCES users(id), -- Initiator
    initiator_name VARCHAR(255) NOT NULL, -- "John Doe" or "System"
    type VARCHAR(255) NOT NULL, -- "inventory.sync"
    status VARCHAR(50) NOT NULL, -- queued, running, completed
    outcome VARCHAR(50) NOT NULL, -- pending, succeeded, partially_succeeded, failed, cancelled
    run_identity_hash VARCHAR(64) NOT NULL, -- SHA256(tenant_id + inputs)
    summary_counts JSONB DEFAULT '{}', -- { success: 10, failed: 2 }
    failure_summary JSONB DEFAULT '[]', -- [{ code: "ERR_TIMEOUT", message: "..." }]
    context JSONB DEFAULT '{}', -- { selection: [...], restore_run_id: 123 }
    started_at TIMESTAMP NULL,
    completed_at TIMESTAMP NULL,
    created_at TIMESTAMP,
    updated_at TIMESTAMP
);

CREATE UNIQUE INDEX operation_runs_active_unique 
ON operation_runs (tenant_id, run_identity_hash) 
WHERE status IN ('queued', 'running');
```

## 4. Risks & Mitigations
-   **Risk**: Desync between `RestoreRun` and `OperationRun`.
    -   **Mitigation**: Use model observers or service-layer wrapping to ensure atomic-like updates, or accept slight eventual consistency (Monitoring might lag ms behind Restore UI).
-   **Risk**: Legacy runs not appearing.
    -   **Mitigation**: We are NOT backfilling legacy runs. Only new runs after deployment will appear in the new Monitoring UI. This is acceptable for "Phase 1".
-   **Risk**: Confusion about `queued` for restore `previewed`.
    -   **Mitigation**: Document that `restore.execute` appears from `previewed` onward and uses `queued/pending` until execution begins; Monitoring remains view-only and links to the restore domain detail.