TenantAtlas/specs/053-unify-runs-monitoring/research.md

Research: Unified Operations Runs + Monitoring Hub (053)

This document resolves Phase 0 open questions and records design choices for Feature 053.

Decisions

1) Canonical run record (Phase 1)

Decision: Reuse the existing bulk_operation_runs / App\Models\BulkOperationRun as the canonical “operation run” record for Phase 1.

Rationale:

• The codebase already uses BulkOperationRun for long-running background work (including Drift generation and Backup Set “Add Policies”).
• It already supports tenant scoping, initiator attribution, counts, and safe failure persistence.
• Avoids a high-risk cross-feature migration before we have proven consistent semantics across modules.

Alternatives considered:

• Create a new generic operation_runs (+ optional operation_run_items) model and migrate all producers to it.
  • Rejected (Phase 1): higher schema + refactor cost, higher coordination risk, and would slow down delivering the Monitoring hub.

2) Monitoring/Operations hub surface

Decision: Implement the Monitoring/Operations hub by evolving the existing Filament BulkOperationRunResource (navigation group/label + filters), rather than creating a new custom monitoring page in Phase 1.

Rationale:

• The resource already provides a tenant-scoped list and a run detail view.
• Small changes deliver high value quickly and reduce risk.

Alternatives considered:

• New “Monitoring → Operations” Filament Page + bespoke table/detail.
  • Rejected (Phase 1): duplicates existing capabilities and increases maintenance.

3) View-only guardrail and viewer roles

Decision: Monitoring/Operations is view-only in Phase 1 and is visible to tenant roles Owner, Manager, Operator, and Readonly. Start/re-run controls remain in the respective feature UIs.

Rationale:

• Adding run management actions implies introducing cancellation semantics, locks, permission matrices, and race handling across producers.
• View-only delivers the primary value (transparency + auditability) without expanding scope.

Alternatives considered:

• Add Rerun / Cancel actions in the hub.
  • Rejected (Phase 1): scope expansion into “run management”.
• Restrict viewing to non-Readonly roles.
  • Rejected: increases “what happened?” support loops; viewing is safe when sanitized.

4) Status semantics and mapping

Decision: Standardize UI-level status semantics as queued → running → (succeeded | partially succeeded | failed) while allowing underlying storage to keep its current status vocabulary.

• partially succeeded = at least one success and at least one failure.
• failed = zero successes (or the run could not proceed).
• BulkOperationRun.status mapping: pending→queued, running→running, completed→succeeded, completed_with_errors→partially succeeded, failed/aborted→failed.

Rationale:

• Keeps the operator-facing meaning consistent and testable without forcing a broad “rename statuses everywhere” refactor.

Alternatives considered:

• Normalize all run status values across all run tables immediately.
  • Rejected (Phase 1): broad blast radius across many features and tests.

5) Failure detail storage

Decision: Persist stable reason codes and short sanitized messages for failures; itemized operations also store a sanitized per-item failures list.

Rationale:

• Operators and support should understand failures without reading server logs.
• Per-item failures avoid rerunning large operations just to identify the affected item.

Alternatives considered:

• Summary-only failure storage.
  • Rejected: loses actionable “which item failed?” detail for itemized runs.
• Logs-only (no persisted failure detail).
  • Rejected: weaker observability and not aligned with “safe, actionable failures”.

6) Idempotency & de-duplication

Decision: Use deterministic idempotency keys and active-run reuse as the primary dedupe mechanism:

• Key builder: App\Support\RunIdempotency::buildKey(...) with stable, sorted context.
• Active-run lookup: reuse when status is active (pending/running).
• Race reduction: rely on the existing partial unique index for active runs and handle collisions by finding and reusing the existing run.

Rationale:

• Aligns with the constitution (“Operations / Run Observability Standard”).
• Durable across restarts and observable in the database.

Alternatives considered:

• Cache-only locks without persisted keys.
  • Rejected: less observable and easier to break across deploys/restarts.

7) Phase 1 producer scope

Decision: Phase 1 adopts the unified monitoring semantics for:

• Drift generation (drift.generate)
• Backup Set “Add Policies” (backup_set.add_policies)

Rationale:

• Both are already using BulkOperationRun and provide immediate value in the Monitoring hub.
• Keeps Phase 1 bounded while proving the pattern across two modules.

Alternatives considered:

• Include every long-running producer in one pass.
  • Rejected (Phase 1): larger blast radius and higher coordination cost.

Notes

• Retention/purge policy for run history should follow existing platform retention controls (defer to planning if changes are required).