107 lines
5.2 KiB
Markdown
107 lines
5.2 KiB
Markdown
# Research: Unified Operations Runs + Monitoring Hub (053)
|
|
|
|
This document resolves Phase 0 open questions and records design choices for Feature 053.
|
|
|
|
## Decisions
|
|
|
|
### 1) Canonical run record (Phase 1)
|
|
|
|
**Decision:** Reuse the existing `bulk_operation_runs` / `App\Models\BulkOperationRun` as the canonical “operation run” record for Phase 1.
|
|
|
|
**Rationale:**
|
|
- The codebase already uses `BulkOperationRun` for long-running background work (including Drift generation and Backup Set “Add Policies”).
|
|
- It already supports tenant scoping, initiator attribution, counts, and safe failure persistence.
|
|
- Avoids a high-risk cross-feature migration before we have proven consistent semantics across modules.
|
|
|
|
**Alternatives considered:**
|
|
- **Create a new generic `operation_runs` (+ optional `operation_run_items`) model** and migrate all producers to it.
|
|
- Rejected (Phase 1): higher schema + refactor cost, higher coordination risk, and would slow down delivering the Monitoring hub.
|
|
|
|
### 2) Monitoring/Operations hub surface
|
|
|
|
**Decision:** Implement the Monitoring/Operations hub by evolving the existing Filament `BulkOperationRunResource` (navigation group/label + filters), rather than creating a new custom monitoring page in Phase 1.
|
|
|
|
**Rationale:**
|
|
- The resource already provides a tenant-scoped list and a run detail view.
|
|
- Small changes deliver high value quickly and reduce risk.
|
|
|
|
**Alternatives considered:**
|
|
- **New “Monitoring → Operations” Filament Page + bespoke table/detail.**
|
|
- Rejected (Phase 1): duplicates existing capabilities and increases maintenance.
|
|
|
|
### 3) View-only guardrail and viewer roles
|
|
|
|
**Decision:** Monitoring/Operations is view-only in Phase 1 and is visible to tenant roles `Owner`, `Manager`, `Operator`, and `Readonly`. Start/re-run controls remain in the respective feature UIs.
|
|
|
|
**Rationale:**
|
|
- Adding run management actions implies introducing cancellation semantics, locks, permission matrices, and race handling across producers.
|
|
- View-only delivers the primary value (transparency + auditability) without expanding scope.
|
|
|
|
**Alternatives considered:**
|
|
- **Add `Rerun` / `Cancel` actions in the hub.**
|
|
- Rejected (Phase 1): scope expansion into “run management”.
|
|
- **Restrict viewing to non-Readonly roles.**
|
|
- Rejected: increases “what happened?” support loops; viewing is safe when sanitized.
|
|
|
|
### 4) Status semantics and mapping
|
|
|
|
**Decision:** Standardize UI-level status semantics as `queued → running → (succeeded | partially succeeded | failed)` while allowing underlying storage to keep its current status vocabulary.
|
|
|
|
- `partially succeeded` = at least one success and at least one failure.
|
|
- `failed` = zero successes (or the run could not proceed).
|
|
- `BulkOperationRun.status` mapping: `pending→queued`, `running→running`, `completed→succeeded`, `completed_with_errors→partially succeeded`, `failed/aborted→failed`.
|
|
|
|
**Rationale:**
|
|
- Keeps the operator-facing meaning consistent and testable without forcing a broad “rename statuses everywhere” refactor.
|
|
|
|
**Alternatives considered:**
|
|
- **Normalize all run status values across all run tables immediately.**
|
|
- Rejected (Phase 1): broad blast radius across many features and tests.
|
|
|
|
### 5) Failure detail storage
|
|
|
|
**Decision:** Persist stable reason codes and short sanitized messages for failures; itemized operations also store a sanitized per-item failures list.
|
|
|
|
**Rationale:**
|
|
- Operators and support should understand failures without reading server logs.
|
|
- Per-item failures avoid rerunning large operations just to identify the affected item.
|
|
|
|
**Alternatives considered:**
|
|
- **Summary-only failure storage.**
|
|
- Rejected: loses actionable “which item failed?” detail for itemized runs.
|
|
- **Logs-only (no persisted failure detail).**
|
|
- Rejected: weaker observability and not aligned with “safe, actionable failures”.
|
|
|
|
### 6) Idempotency & de-duplication
|
|
|
|
**Decision:** Use deterministic idempotency keys and active-run reuse as the primary dedupe mechanism:
|
|
- Key builder: `App\Support\RunIdempotency::buildKey(...)` with stable, sorted context.
|
|
- Active-run lookup: reuse when status is active (`pending`/`running`).
|
|
- Race reduction: rely on the existing partial unique index for active runs and handle collisions by finding and reusing the existing run.
|
|
|
|
**Rationale:**
|
|
- Aligns with the constitution (“Operations / Run Observability Standard”).
|
|
- Durable across restarts and observable in the database.
|
|
|
|
**Alternatives considered:**
|
|
- **Cache-only locks without persisted keys.**
|
|
- Rejected: less observable and easier to break across deploys/restarts.
|
|
|
|
### 7) Phase 1 producer scope
|
|
|
|
**Decision:** Phase 1 adopts the unified monitoring semantics for:
|
|
- Drift generation (`drift.generate`)
|
|
- Backup Set “Add Policies” (`backup_set.add_policies`)
|
|
|
|
**Rationale:**
|
|
- Both are already using `BulkOperationRun` and provide immediate value in the Monitoring hub.
|
|
- Keeps Phase 1 bounded while proving the pattern across two modules.
|
|
|
|
**Alternatives considered:**
|
|
- **Include every long-running producer in one pass.**
|
|
- Rejected (Phase 1): larger blast radius and higher coordination cost.
|
|
|
|
## Notes
|
|
|
|
- Retention/purge policy for run history should follow existing platform retention controls (defer to planning if changes are required).
|