TenantAtlas/specs/094-assignment-ops-observability-hardening/spec.md

# Feature Specification: Assignment Operations Observability Hardening

**Feature Branch**: `094-assignment-ops-observability-hardening`  
**Created**: 2026-02-14  
**Status**: Draft  
**Input**: User description: "Harden assignment operation observability and close remaining authorization inconsistencies so operations are fully traceable, diagnosable, and correctly access-controlled."

## Spec Scope Fields *(mandatory)*

- **Scope**: canonical-view
- **Primary Routes**:
  - Admin Monitoring → Operations (list + detail views)
  - Provider Connections list
  - Backup Sets list
  - Restore Runs list
  - Backup Items relationship view under a backup set
- **Data Ownership**: operational activity records must be scoped to the correct workspace and tenant context where applicable; no new domain entities are introduced.
- **RBAC**: all affected admin surfaces require workspace membership; actions that change configuration or trigger restores require explicit permissions as defined in the central capability registry.

### Canonical-view Constraints

- **Default behavior when tenant-context is active**: Monitoring → Operations defaults to showing operational records for the currently selected tenant context (if one is selected).
- **Entitlement checks**: Monitoring → Operations MUST not reveal tenant-owned operational records unless the actor is entitled to that tenant scope within the active workspace context.

## Clarifications

### Session 2026-02-14

- Q: For FR-005 / SC-004, what should “identity scope” mean for deduping “active” runs? → A: Dedupe per tenant and operation target/scope.
- Q: For FR-003 (“summary counters”) what exact semantics should total / processed / failed follow? → A: total = items attempted; processed = succeeded; failed = failed.
- Q: For FR-004 (“stable failure code”), which convention should we standardize on for assignment fetch/restore runs? → A: Operation-specific code namespaces + normalized reason_code for the cause.
- Q: For FR-006 (identity scope = tenant + operation type + target/scope), which identifiers should define “target/scope” for these two operations? → A: Fetch targets the backup item (or policy version) identifier; restore targets the restore run identifier.
- Q: For US1 / FR-002 (“restore is observable/auditable”), what audit log granularity do you want for assignment restore? → A: One audit log entry per assignment restore execution (per restore run).

## User Scenarios & Testing *(mandatory)*

### User Story 1 — Observe assignment operations end-to-end (Priority: P1)

Workspace administrators need to see assignment-related operations (both read-only fetch and destructive restore) in Monitoring so they can confirm what ran, what changed, and why something failed, without relying on server logs.

**Why this priority**: Assignment restore is high-risk; missing visibility creates operational and audit gaps.

**Independent Test**: Trigger both an assignment fetch and an assignment restore; verify each produces a monitoring-visible run record with correct lifecycle and failure details.

**Acceptance Scenarios**:

1. **Given** an administrator triggers an assignment fetch, **When** the operation starts and completes, **Then** Monitoring shows a run record with start/end timestamps, final outcome, and summary counters.
2. **Given** an administrator triggers an assignment restore, **When** the operation starts and completes, **Then** Monitoring shows a run record including a clear indication that it was a change-making operation.
  - And exactly one audit log entry is written for the restore execution.
3. **Given** an assignment fetch or restore fails due to an external dependency error, **When** the run completes, **Then** Monitoring shows a stable failure code and a sanitized, user-readable message.
4. **Given** the same assignment operation is triggered multiple times concurrently for the same tenant and scope, **When** the system creates tracking records, **Then** the admin sees a single “active” run per identity (or an equivalent deduped representation).

---

### User Story 2 — Enforce correct access control semantics on affected admin surfaces (Priority: P2)

Workspace administrators and platform operators must not be able to cross authentication “planes” accidentally, and the admin UI must not expose bypasses that let users initiate sensitive actions without authorization.

**Why this priority**: Prevents cross-plane leakage and closes known authorization inconsistencies.

**Independent Test**: Attempt access with the wrong authentication plane and with insufficient permissions; verify outcomes are deny-as-not-found (404) or forbidden (403) per policy.

**Acceptance Scenarios**:

1. **Given** a user is authenticated in a different auth plane, **When** they attempt to access workspace-scoped admin routes, **Then** the response is deny-as-not-found (404).
2. **Given** a user is not a member of the workspace, **When** they attempt to view backup items under a backup set, **Then** the response is deny-as-not-found (404) and does not reveal record existence.
3. **Given** a user is a workspace member but lacks the required permission, **When** they attempt a protected action (such as managing provider connections), **Then** the response is forbidden (403).

---

### User Story 3 — Validate assignment operations safely in non-production contexts (Priority: P3)

Platform engineers need to validate that assignment operations behave correctly without requiring live external dependencies, so regressions can be caught early.

**Why this priority**: Improves reliability and reduces the risk of shipping changes that only fail when external services are slow/unavailable.

**Independent Test**: Run automated tests that simulate both successful and failing external interactions; verify monitoring records and authorization behaviors.

**Acceptance Scenarios**:

1. **Given** external interactions are simulated as “successful”, **When** the operation runs, **Then** the run is marked successful and includes expected summary counters.
2. **Given** external interactions are simulated as “failing”, **When** the operation runs, **Then** the run is marked failed with a stable failure code and sanitized message.

### Edge Cases

- External dependency timeouts, throttling, or transient failures.
- Retries and duplicate dispatches (ensure tracking remains coherent and non-spammy).
- Missing or inconsistent tenant/workspace context (must fail safely, not leak).
- Partial completion (some items processed, some failed): counters and failure details must remain interpretable.

## Requirements *(mandatory)*

### Functional Requirements

- **FR-001**: The system MUST create and maintain an operational tracking record (OperationRun) for every assignment fetch operation execution.
- **FR-002**: The system MUST create and maintain an operational tracking record (OperationRun) for every assignment restore operation execution.
- **FR-003**: Tracking records MUST include lifecycle state (queued/running/completed), timestamps, outcome, and summary counters sufficient to understand progress and results.
  - Counter semantics: `total` = items attempted, `processed` = succeeded, `failed` = failed.
- **FR-004**: Failed runs MUST include a stable failure code and a sanitized, user-readable failure message.
  - Failure convention: use operation-specific `code` namespaces for the run, and store the underlying cause as a normalized `reason_code`.
- **FR-005**: The system MUST prevent duplicate “active” runs for the same tenant and identity scope (tenant + operation type + operation target/scope), or otherwise present a deduped representation that avoids operator confusion.
- **FR-006**: Identity scope MUST be defined as:
  - For assignment fetch operations: tenant + operation type + backup item identifier (or equivalent policy-version identifier).
  - For assignment restore operations: tenant + operation type + restore run identifier.
- **FR-007**: Monitoring pages MUST render using persisted operational data only (no outbound calls during page render).
- **FR-008**: Cross-plane access MUST be deny-as-not-found (404) on affected routes.
- **FR-009**: Authorization MUST follow consistent semantics:
  - non-member / not entitled → 404 (deny-as-not-found)
  - member without required permission → 403
- **FR-010**: Any action that can change configuration or trigger a restore MUST be server-authorized; UI visibility MUST NOT be treated as authorization.
- **FR-011**: Sensitive or destructive-like actions MUST require explicit confirmation.
- **FR-012**: Each assignment restore execution MUST write exactly one audit log entry for the restore run.

### Dependencies & Assumptions

- Assignment fetch and assignment restore operations can be triggered by administrators or scheduled/queued execution paths.
- Monitoring users have access to the Operations area only when they have appropriate workspace membership and permissions.
- External dependency failures may occur and must be represented consistently (stable failure codes + sanitized messages).

## UI Action Matrix *(mandatory when the admin UI is changed)*

| Surface | Location | Header Actions | Inspect Affordance (List/Table) | Row Actions (max 2 visible) | Bulk Actions (grouped) | Empty-State CTA(s) | View Header Actions | Create/Edit Save+Cancel | Audit log? | Notes / Exemptions |
|---|---|---|---|---|---|---|---|---|---|---|
| Provider Connections | Admin → Provider Connections | Create connection (permission-gated) | View connection details | Edit / Delete (permission-gated; destructive confirms) | None | Create connection | N/A | Save + Cancel | Yes | Ensures no authorization bypass exists on header/empty-state CTAs. |
| Backup Sets | Admin → Backups | None (unchanged) | View backup set | Actions unchanged | None | None | N/A | N/A | N/A | Only enforcement helper consistency is affected. |
| Restore Runs | Admin → Restores | None (unchanged) | View restore run | Actions unchanged | None | None | N/A | N/A | Yes | Restore operations must be observable via Monitoring. |
| Backup Items | Under a backup set | None | View backup item details | Actions unchanged | None | None | N/A | N/A | N/A | Membership/404 checks must occur before capability/403 checks. |
| Monitoring → Operations | Admin → Monitoring | None | View operation run details | None | None | None | N/A | N/A | Yes | Read-only view; must not call external services during render. |

## Success Criteria *(mandatory)*

### Measurable Outcomes

- **SC-001**: 100% of assignment fetch and assignment restore executions appear in Monitoring with a completed outcome (success or failure) and timestamps.
- **SC-002**: For failed runs, operators can identify a stable failure code and a readable failure message from Monitoring within 60 seconds, without checking server logs.
- **SC-003**: Automated tests verify deny-as-not-found (404) vs forbidden (403) semantics for the affected surfaces.
- **SC-004**: Duplicate active-run confusion is eliminated: repeated triggers produce a single active run per identity scope (tenant + operation type + operation target/scope) or equivalent deduped visibility.