From 1fa15b4db23406326073fe7da465989ee3bfa181 Mon Sep 17 00:00:00 2001 From: Ahmed Darrazi Date: Mon, 22 Dec 2025 01:07:03 +0100 Subject: [PATCH] spec: Feature 004 & 005 with production-tested Graph API strategies Feature 004 (Assignments & Scope Tags): - Use fallback strategy for assignments read (direct + $expand) - Use POST /directoryObjects/getByIds for stable group resolution - POST /assign only (not PATCH) for assignments write - Handle 204 No Content responses Feature 005 (Bulk Operations): - Policies: Local delete only (ignored_at flag, no Graph DELETE) - Policy Versions: Eligibility checks + retention policy - BulkOperationRun model for progress tracking - Livewire polling for UI updates (not automatic) - Chunked processing + circuit breaker (abort >50% fail) - array $ids in Job constructor (not Collection) --- specs/004-assignments-scope-tags/spec.md | 472 +++++++++++++++++ specs/005-bulk-operations/spec.md | 617 +++++++++++++++++++++++ 2 files changed, 1089 insertions(+) create mode 100644 specs/004-assignments-scope-tags/spec.md create mode 100644 specs/005-bulk-operations/spec.md diff --git a/specs/004-assignments-scope-tags/spec.md b/specs/004-assignments-scope-tags/spec.md new file mode 100644 index 0000000..58c405c --- /dev/null +++ b/specs/004-assignments-scope-tags/spec.md @@ -0,0 +1,472 @@ +# Feature 004: Assignments & Scope Tags for Settings Catalog Policies + +## Overview +Extend backup and restore functionality to include **Assignments** (group/user/device targeting) and **Scope Tags** for Settings Catalog policies. This ensures complete policy state capture and enables cross-tenant migrations with group mapping. + +## Problem Statement +Currently, TenantPilot backs up only the Settings Catalog policy configuration itself. However, **Assignments** (which groups/users/devices the policy applies to) and **Scope Tags** (RBAC-based policy visibility) are critical metadata that make a policy actionable. + +**Without this feature:** +- Restored policies have no assignments → must be manually configured +- Cross-tenant migrations lose targeting context +- Scope Tags are not preserved → RBAC-scoped admins may lose access +- Backup previews don't show assignment counts or scope + +**With this feature:** +- Complete policy state backup (config + assignments + scope tags) +- Cross-tenant restore with intelligent group mapping +- Clear preview/diff of assignment changes +- Optional inclusion (lightweight backups possible) + +## Goals +- **Primary**: Backup and restore assignments for Settings Catalog policies +- **Secondary**: Include Scope Tags in backup metadata +- **Tertiary**: Group mapping wizard for cross-tenant restores +- **Non-Goal**: Assignments for non-Settings Catalog types (future features 006-009) + +## Scope +- **Policy Types**: `settingsCatalogPolicy` only (initially) +- **Graph Endpoints**: + - GET `/deviceManagement/configurationPolicies/{id}/assignments` + - POST/PATCH `/deviceManagement/configurationPolicies/{id}/assign` + - GET `/deviceManagement/roleScopeTags` (for reference data) +- **Backup Behavior**: Optional (checkbox "Include Assignments & Scope Tags") +- **Restore Behavior**: With group mapping UI for unresolved group IDs + +--- + +## User Stories + +### User Story 1 - Backup with Assignments & Scope Tags (Priority: P1) + +**As an admin**, I want to optionally include assignments and scope tags when backing up Settings Catalog policies, so that I have complete policy state for migration or disaster recovery. + +**Acceptance Criteria:** +1. **Given** I create a new Backup Set for Settings Catalog policies, + **When** I enable the checkbox "Include Assignments & Scope Tags", + **Then** the backup captures: + - Assignment list (groups, users, devices with include/exclude mode) + - Scope Tag IDs referenced by the policy + - Metadata about assignment count and scope tag names + +2. **Given** I view a Backup Set with assignments included, + **When** I expand a Backup Item detail, + **Then** I see: + - "Assignments: 3 groups, 2 users" summary + - "Scope Tags: Default, HR-Admins" list + - JSON tab with full assignment payload + +3. **Given** I create a Backup Set without enabling the checkbox, + **When** the backup completes, + **Then** assignments and scope tags are NOT captured (payload-only backup) + +--- + +### User Story 2 - Policy View with Assignments Tab (Priority: P1) + +**As an admin**, I want to see a policy's current assignments and scope tags in the Policy View, so I understand its targeting and visibility. + +**Acceptance Criteria:** +1. **Given** I view a Settings Catalog policy, + **When** I navigate to the "Assignments" tab, + **Then** I see: + - Table with columns: Type (Group/User/Device), Name, Mode (Include/Exclude), ID + - "Scope Tags" section showing: Default, HR-Admins (editable IDs) + - "Not assigned" message if no assignments exist + +2. **Given** a policy has 10 assignments, + **When** I filter by "Include only" or "Exclude only", + **Then** the table filters accordingly + +3. **Given** assignments include deleted groups (orphaned IDs), + **When** I view the assignments tab, + **Then** orphaned entries show as "Unknown Group (ID: abc-123)" with warning badge + +--- + +### User Story 3 - Restore with Group Mapping (Priority: P1) + +**As an admin**, I want to map source tenant groups to target tenant groups during restore, so I can migrate policies across tenants without manual re-assignment. + +**Acceptance Criteria:** +1. **Given** I restore a Backup Item with assignments to a different tenant, + **When** the restore preview detects unresolved group IDs, + **Then** the wizard shows a "Group Mapping" step with: + - Source group name and ID + - Target tenant groups dropdown (searchable) + - "Skip assignment" checkbox per group + +2. **Given** I map 3 source groups to target groups, + **When** I confirm the restore, + **Then** the restored policy has assignments pointing to the mapped target groups + +3. **Given** I choose "Skip assignment" for 2 groups, + **When** the restore completes, + **Then** those 2 assignments are NOT created (only mapped groups restored) + +4. **Given** all source groups exist in target tenant (same IDs), + **When** the restore runs, + **Then** the Group Mapping step is skipped (auto-matched) + +--- + +### User Story 4 - Restore Preview with Assignment Diff (Priority: P2) + +**As an admin**, I want to see assignment changes in the restore preview, so I know what will be modified before executing. + +**Acceptance Criteria:** +1. **Given** I preview a restore that includes assignments, + **When** the target policy has different assignments, + **Then** the preview shows: + - "Assignments: 3 will be added, 1 removed, 2 unchanged" + - Expandable diff: Added (green), Removed (red), Unchanged (gray) + +2. **Given** the target policy has no existing assignments, + **When** I preview the restore, + **Then** the preview shows "Assignments: 5 will be created" + +3. **Given** I restore a backup without assignments, + **When** I preview the restore, + **Then** the assignment section shows "Not included in backup" + +--- + +## Functional Requirements + +### Backup & Storage + +**FR-004.1**: System MUST provide a checkbox "Include Assignments & Scope Tags" on the Backup Set creation form (default: unchecked). + +**FR-004.2**: When assignments are included, system MUST fetch assignments using fallback strategy: +1. Try: `/deviceManagement/configurationPolicies/{id}/assignments` +2. If empty/fails: Try `$expand=assignments` on policy fetch +3. Store: + - Assignment array (each with: `target` object, `id`, `intent`, filters) + - Extracted metadata: group names (resolved via `/directoryObjects/getByIds`), user UPNs, device IDs + - Warning flags for orphaned IDs + - Fallback flag: `assignments_fetch_method` (direct | expand | failed) + +**FR-004.3**: System MUST store Scope Tag IDs in backup metadata (from policy payload `roleScopeTagIds` field). + +**FR-004.4**: Backup Item `metadata` JSONB field MUST include: +```json +{ + "assignment_count": 5, + "scope_tag_ids": ["0", "abc-123"], + "scope_tag_names": ["Default", "HR-Admins"], + "has_orphaned_assignments": false +} +``` + +**FR-004.5**: System MUST gracefully handle Graph API failures when fetching assignments (log warning, continue backup with flag `assignments_fetch_failed: true`). + +### UI Display + +**FR-004.6**: Policy View MUST show an "Assignments" tab for Settings Catalog policies displaying: +- Assignments table (type, name, mode, ID) +- Scope Tags section +- Empty state if no assignments + +**FR-004.7**: Backup Item detail view MUST show assignment count and scope tag names in metadata summary. + +**FR-004.8**: System MUST render orphaned group IDs as "Unknown Group (ID: {id})" with warning icon. + +### Restore with Group Mapping + +**FR-004.9**: Restore preview MUST detect unresolved group IDs by calling POST `/directoryObjects/getByIds` with batch of group IDs extracted from source assignments (types: ["group"]). Missing IDs = unresolved. + +**FR-004.10**: When unresolved groups exist, system MUST inject a "Group Mapping" step in the restore wizard showing: +- Source group (name from backup metadata or ID if name unavailable) +- Target group dropdown (searchable, populated from target tenant) +- "Skip" checkbox + +**FR-004.11**: System MUST persist group mapping selections in RestoreRun metadata for audit and rerun purposes. + +**FR-004.12**: When restoring assignments, system MUST: +1. Replace source group IDs with mapped target group IDs +2. Skip assignments marked "Skip" +3. Preserve include/exclude intent and filters +4. Call POST `/deviceManagement/configurationPolicies/{id}/assign` (not PATCH) with complete mapped assignments array (replaces all assignments atomically) +5. Handle 204 No Content or 200 OK as success +6. Log Graph request-id and client-request-id on failure + +**FR-004.13**: System MUST handle assignment restore failures gracefully: +- Log per-assignment outcome (success/skip/failure) +- Continue with remaining assignments +- Report final status: "3 of 5 assignments restored" + +**FR-004.14**: System MUST write audit log entries: +- `backup.assignments.included` (when checkbox enabled) +- `restore.group_mapping.applied` (with mapping details) +- `restore.assignment.created` (per assignment) +- `restore.assignment.skipped` (per skipped) + +### Scope Tags + +**FR-004.15**: System MUST extract Scope Tag IDs from policy payload's `roleScopeTagIds` array during backup. + +**FR-004.16**: System MUST resolve Scope Tag names via `/deviceManagement/roleScopeTags` (with caching, TTL 1 hour). + +**FR-004.17**: During restore, system SHOULD preserve Scope Tag IDs if they exist in target tenant, or: +- Log warning if Scope Tag ID doesn't exist in target +- Allow policy creation to proceed (Graph API default behavior) + +**FR-004.18**: Restore preview MUST show Scope Tag diff: "Scope Tags: 2 matched, 1 not found in target tenant". + +--- + +## Non-Functional Requirements + +**NFR-004.1**: Assignment fetching MUST NOT block backup creation (async or fail-soft). + +**NFR-004.2**: Group mapping UI MUST support search/filter for tenants with 500+ groups. + +**NFR-004.3**: Assignment restore MUST batch API calls (if Graph supports batch, else sequential with 100ms delay). + +**NFR-004.4**: System MUST cache target tenant group list for 5 minutes during restore wizard session. + +--- + +## Data Model Changes + +### Migration: `backup_items` table extension + +```php +Schema::table('backup_items', function (Blueprint $table) { + $table->json('assignments')->nullable()->after('metadata'); + // stores: [{target:{...}, id, intent, filters}, ...] +}); +``` + +### Migration: `restore_runs` table extension + +```php +Schema::table('restore_runs', function (Blueprint $table) { + $table->json('group_mapping')->nullable()->after('results'); + // stores: {"source-group-id": "target-group-id", ...} +}); +``` + +### `backup_items.metadata` JSONB schema + +```json +{ + "assignment_count": 5, + "scope_tag_ids": ["0", "123"], + "scope_tag_names": ["Default", "HR"], + "has_orphaned_assignments": false, + "assignments_fetch_failed": false +} +``` + +--- + +## Graph API Integration + +### Endpoints to Add (Production-Tested Strategies) + +1. **GET Assignments (with Fallback Strategy)** + - **Primary**: `/deviceManagement/configurationPolicies/{id}/assignments` + - Returns: `{ value: [assignment objects] }` + - Contract: `type_family: [#microsoft.graph.deviceManagementConfigurationPolicyAssignment]` + - **Fallback** (if primary fails/returns empty): + - `/deviceManagement/configurationPolicies?$expand=assignments&$filter=id eq '{id}'` + - Client-side filter to extract assignments + - **Reason**: Known Graph API quirks with assignment expansion on certain template families + +2. **POST** `/deviceManagement/configurationPolicies/{id}/assign` (POST only, NOT PATCH) + - Body: `{ assignments: [assignment objects] }` + - Returns: 204 No Content or 200 OK + - **Note**: This is an action endpoint, replaces entire assignments array + - Example payload: + ```json + { + "assignments": [ + { + "id": "00000000-0000-0000-0000-000000000000", + "target": { + "@odata.type": "#microsoft.graph.groupAssignmentTarget", + "groupId": "abc-123-def" + }, + "intent": "apply" + } + ] + } + ``` + +3. **POST** `/directoryObjects/getByIds` (Stable Group Resolution) + - Body: `{ "ids": ["id1", "id2"], "types": ["group"] }` + - Returns: `{ value: [{ id, displayName, ... }] }` + - **Reason**: More stable than `$filter=id in (...)` which can fail with advanced query requirements + - Example: + ```json + { + "ids": ["abc-123", "def-456"], + "types": ["group"] + } + ``` + +4. **GET** `/deviceManagement/roleScopeTags?$select=id,displayName` + - For Scope Tag resolution (cache 1 hour) + - Scope Tag IDs also available in policy payload's `roleScopeTagIds` array + +### Graph Contract Updates + +Add to `config/graph_contracts.php`: + +```php +'settingsCatalogPolicy' => [ + // ... existing + 'assignments_path' => '/deviceManagement/configurationPolicies/{id}/assignments', + 'assign_method' => 'POST', + 'assign_path' => '/deviceManagement/configurationPolicies/{id}/assign', + 'supports_scope_tags' => true, +], +``` + +--- + +## UI Mockups (Wireframe Descriptions) + +### Policy View - Assignments Tab + +``` +[General] [Settings] [Assignments] [JSON] + +Assignments (5) +┌─────────────────────────────────────────────────┐ +│ Type │ Name │ Mode │ ID │ +├─────────┼───────────────────┼─────────┼─────────┤ +│ Group │ All Users │ Include │ abc-123 │ +│ Group │ Contractors │ Exclude │ def-456 │ +│ User │ john@contoso.com │ Include │ ghi-789 │ +└─────────────────────────────────────────────────┘ + +Scope Tags (2) + • Default (ID: 0) + • HR-Admins (ID: 123) +``` + +### Backup Creation - Checkbox + +``` +Create Backup Set +───────────────── +Select Policies: [Settings Catalog: 15 selected] + +☑ Include Assignments & Scope Tags + Captures group/user targeting and RBAC scope. + Adds ~2-5 KB per policy with assignments. + +[Cancel] [Create Backup] +``` + +### Restore Wizard - Group Mapping Step + +``` +Restore Preview > Group Mapping > Confirm + +Group Mapping Required +Some groups from source tenant don't exist in target tenant. + +┌────────────────────────────────────────────────────────┐ +│ Source Group │ Target Group │ Action │ +├───────────────────────┼───────────────────────┼────────┤ +│ All Users (abc-123) │ [Select target group] │ ☐ Skip │ +│ HR Team (def-456) │ HR Department │ │ +│ Contractors (ghi-789) │ [Select target group] │ ☑ Skip │ +└────────────────────────────────────────────────────────┘ + +[Back] [Continue] +``` + +--- + +## Testing Strategy + +### Unit Tests +- `AssignmentFetcherTest`: Mock Graph responses, test parsing +- `GroupMapperTest`: Test ID resolution, mapping logic +- `ScopeTagResolverTest`: Test caching, name resolution + +### Feature Tests +- `BackupWithAssignmentsTest`: E2E backup creation with checkbox +- `PolicyViewAssignmentsTabTest`: UI rendering, orphaned IDs +- `RestoreGroupMappingTest`: Wizard flow, mapping persistence +- `RestoreAssignmentApplicationTest`: Graph API calls, outcomes + +### Manual QA +- Create backup with/without assignments checkbox +- Restore to same tenant (auto-match groups) +- Restore to different tenant (group mapping wizard) +- Handle orphaned group IDs gracefully + +--- + +## Rollout Plan + +### Phase 1: Backup with Assignments (MVP) +- Add checkbox to Backup form +- Fetch assignments from Graph +- Store in `backup_items.assignments` +- Display in Policy View (read-only) +- **Duration**: ~8-12 hours + +### Phase 2: Restore with Group Mapping +- Add Group Mapping wizard step +- Implement ID resolution +- Apply assignments on restore +- **Duration**: ~12-16 hours + +### Phase 3: Scope Tags +- Resolve Scope Tag names +- Display in UI +- Handle restore warnings +- **Duration**: ~4-6 hours + +### Phase 4: Future Extensions +- Feature 006: Extend to `compliancePolicy` +- Feature 007: Extend to `deviceConfiguration` +- Feature 008: Extend to Conditional Access +- Feature 009: Assignment analytics/reporting + +--- + +## Dependencies +- Feature 001: Backup/Restore core (✅ complete) +- Graph Contract Registry (✅ complete) +- Filament multi-step forms (built-in) + +## Risks & Mitigations + +| Risk | Mitigation | +|------|------------| +| Graph API assignments endpoint slow/fails | Async fetch, fail-soft with warning | +| Target tenant has 1000+ groups | Searchable dropdown with pagination | +| Group IDs change across tenants | Group name-based matching fallback | +| Scope Tag IDs don't exist in target | Log warning, allow policy creation | + +--- + +## Success Criteria + +1. ✅ Backup checkbox functional, assignments captured +2. ✅ Policy View shows assignments tab with accurate data +3. ✅ Group Mapping wizard handles 100+ groups smoothly +4. ✅ Restore applies assignments with 90%+ success rate +5. ✅ Audit logs record all mapping decisions +6. ✅ Tests achieve 85%+ coverage for new code + +--- + +## Open Questions +1. Should we support "smart matching" (group name similarity) for group mapping? +2. How to handle dynamic groups (membership rules) - copy rules or skip? +3. Should Scope Tag warnings block restore or just warn? + +--- + +**Status**: Draft for Review +**Created**: 2025-12-22 +**Author**: AI + Ahmed +**Next Steps**: Review → Plan → Tasks diff --git a/specs/005-bulk-operations/spec.md b/specs/005-bulk-operations/spec.md new file mode 100644 index 0000000..1c3cc37 --- /dev/null +++ b/specs/005-bulk-operations/spec.md @@ -0,0 +1,617 @@ +# Feature 005: Bulk Operations for Resource Management + +## Overview +Enable efficient bulk operations across TenantPilot's main resources (Policies, Policy Versions, Backup Sets, Restore Runs) to improve admin productivity and reduce repetitive actions. + +## Problem Statement +Currently, admins must perform actions one-by-one on individual resources: +- Deleting 20 old Policy Versions = 20 clicks + confirmations +- Exporting 50 Policies to a Backup = 50 manual selections +- Cleaning up 30 failed Restore Runs = 30 delete actions + +**This is tedious, error-prone, and time-consuming.** + +**With bulk operations:** +- Select multiple items → single action → confirm → done +- Clear audit trail (one bulk action = one audit event + per-item outcomes) +- Progress notifications for long-running operations +- Consistent UX across all resources + +## Goals +- **Primary**: Implement bulk delete, bulk export, bulk restore (soft delete) for main resources +- **Secondary**: Safety gates (confirmation dialogs, type-to-confirm for destructive ops) +- **Tertiary**: Queue-based processing for large batches with progress tracking +- **Non-Goal**: Bulk edit/update (too complex, deferred to future feature) + +--- + +## User Stories + +### User Story 1 - Bulk Delete Policies (Priority: P1) + +**As an admin**, I want to soft-delete multiple policies **locally in TenantPilot** at once, so I can clean up outdated or test policies efficiently. + +**Important**: This action marks policies as deleted locally, does NOT delete them in Intune. Policies are flagged as `ignored_at` to prevent re-sync. + +**Acceptance Criteria:** +1. **Given** I select 15 policies in the Policies table, + **When** I click "Delete (Local)" in the bulk actions menu, + **Then** a confirmation dialog appears: "Delete 15 policies locally? They will be hidden from listings and ignored in sync." + +2. **Given** I confirm the bulk delete, + **When** the operation completes, + **Then**: + - All 15 policies are flagged (`ignored_at` timestamp set, optionally `deleted_at`) + - A success notification shows: "Deleted 15 policies locally" + - An audit log entry `policies.bulk_deleted_local` is created with policy IDs + - Policies remain in Intune (unchanged) + +3. **Given** I bulk-delete 50 policies, + **When** the operation runs, + **Then** it processes asynchronously via queue (job) with progress notification + +4. **Given** I lack `policies.delete` permission, + **When** I try to bulk-delete, + **Then** the bulk action is disabled/hidden (same permission model as single delete) + +--- + +### User Story 2 - Bulk Export Policies to Backup (Priority: P1) + +**As an admin**, I want to export multiple policies to a new Backup Set in one action, so I can quickly snapshot a subset of policies. + +**Acceptance Criteria:** +1. **Given** I select 25 policies, + **When** I click "Export to Backup", + **Then** a dialog prompts: "Backup Set Name" + "Include Assignments?" checkbox + +2. **Given** I confirm the export, + **When** the backup job runs, + **Then**: + - A new Backup Set is created + - 25 Backup Items are captured (one per policy) + - Progress notification: "Backing up... 10/25" + - Final notification: "Backup Set 'Production Snapshot' created (25 items)" + +3. **Given** 3 of 25 policies fail to backup (Graph error), + **When** the job completes, + **Then**: + - 22 items succeed, 3 fail + - Notification: "Backup completed: 22 succeeded, 3 failed" + - Audit log records per-item outcomes + +--- + +### User Story 3 - Bulk Delete Policy Versions (Priority: P2) + +**As an admin**, I want to bulk-delete old policy versions to free up database space, respecting retention policies. + +**Important**: Policy Versions are immutable snapshots. Deletion only allowed if version is NOT referenced (no active Backup Items, Restore Runs, or audit trails) and meets retention threshold (e.g., >90 days old). + +**Acceptance Criteria:** +1. **Given** I select 30 policy versions older than 90 days, + **When** I click "Delete", + **Then** confirmation dialog: "Delete 30 policy versions? This is permanent and cannot be undone." + +2. **Given** I confirm, + **When** the operation completes, + **Then**: + - System checks each version: is_current=false + not referenced + age >90 days + - Eligible versions are hard-deleted + - Ineligible versions are skipped with reason (e.g., "Referenced by Backup Set ID 5") + - Success notification: "Deleted 28 policy versions (2 skipped)" + - Audit log: `policy_versions.bulk_pruned` with version IDs + skip reasons + +3. **Given** I lack `policy_versions.prune` permission, + **When** I try to bulk-delete, + **Then** the bulk action is hidden + +--- + +### User Story 4 - Bulk Delete Restore Runs (Priority: P2) + +**As an admin**, I want to bulk-delete completed or failed Restore Runs to declutter the history. + +**Acceptance Criteria:** +1. **Given** I select 20 restore runs (status: completed/failed), + **When** I click "Delete", + **Then** confirmation: "Delete 20 restore runs? Historical data will be removed." + +2. **Given** I confirm, + **When** the operation completes, + **Then**: + - 20 restore runs are soft-deleted + - Notification: "Deleted 20 restore runs" + - Audit log: `restore_runs.bulk_deleted` + +3. **Given** I select restore runs with mixed statuses (running + completed), + **When** I attempt bulk delete, + **Then** only completed/failed runs are deleted (running ones skipped with warning) + +--- + +### User Story 5 - Bulk Delete with Type-to-Confirm (Priority: P1) + +**As an admin**, I want extra confirmation for large destructive operations, so I don't accidentally delete important data. + +**Acceptance Criteria:** +1. **Given** I bulk-delete ≥20 items, + **When** the confirmation dialog appears, + **Then** I must type "DELETE" in a text field to enable the confirm button + +2. **Given** I type an incorrect word (e.g., "delete" lowercase), + **When** I try to confirm, + **Then** the button remains disabled with error: "Type DELETE to confirm" + +3. **Given** I type "DELETE" correctly, + **When** I click confirm, + **Then** the bulk operation proceeds + +--- + +### User Story 6 - Bulk Operation Progress Tracking (Priority: P2) + +**As an admin**, I want to see real-time progress for bulk operations, so I know the system is working. + +**Acceptance Criteria:** +1. **Given** I bulk-delete 100 policies, + **When** the job starts, + **Then** a Filament notification shows: "Deleting policies... 0/100" + +2. **Given** the job processes items, + **When** progress updates, + **Then** the notification updates every 5 seconds: "Deleting... 45/100" + +3. **Given** the job completes, + **When** all items are processed, + **Then**: + - Final notification: "Deleted 98 policies (2 failed)" + - Clickable link: "View details" → opens audit log entry + +--- + +## Functional Requirements + +### General Bulk Operations + +**FR-005.1**: System MUST provide bulk action checkboxes on table rows for: +- Policies +- Policy Versions +- Backup Sets +- Restore Runs + +**FR-005.2**: Bulk actions menu MUST appear when ≥1 item is selected, showing: +- Action name (e.g., "Delete") +- Count badge (e.g., "3 selected") +- Disabled state if user lacks permission + +**FR-005.3**: System MUST enforce same permissions for bulk actions as single actions (e.g., `policies.delete` for bulk delete). + +**FR-005.4**: Bulk operations processing ≥20 items MUST run via Laravel Queue (async job) using Bus::batch() or chunked processing (batches of 10-20 items). + +**FR-005.4a**: System MUST create a `bulk_operation_runs` table to track progress: +```php +Schema::create('bulk_operation_runs', function (Blueprint $table) { + $table->id(); + $table->foreignId('tenant_id')->constrained(); + $table->foreignId('user_id')->constrained(); + $table->string('resource'); // 'policies', 'policy_versions', etc. + $table->string('action'); // 'delete', 'export', etc. + $table->string('status'); // 'running', 'completed', 'failed', 'aborted' + $table->integer('total_items'); + $table->integer('processed_items')->default(0); + $table->integer('succeeded')->default(0); + $table->integer('failed')->default(0); + $table->integer('skipped')->default(0); + $table->json('item_ids'); // array of IDs + $table->json('failures')->nullable(); // [{id, reason}, ...] + $table->foreignId('audit_log_id')->nullable()->constrained(); + $table->timestamps(); +}); + +**FR-005.5**: Bulk operations <20 items MAY run synchronously (immediate feedback). + +### Confirmation Dialogs + +**FR-005.6**: Confirmation dialog MUST show: +- Action description: "Delete 15 policies?" +- Impact warning: "This moves them to trash." or "This is permanent." +- Item count badge +- Cancel/Confirm buttons + +**FR-005.7**: For destructive operations with ≥20 items, dialog MUST require typing "DELETE" (case-sensitive) to enable confirm button. + +**FR-005.8**: For non-destructive operations (export, restore), typing confirmation is NOT required. + +### Audit Logging + +**FR-005.9**: System MUST create one audit log entry per bulk operation with: +- Event type: `{resource}.bulk_{action}` (e.g., `policies.bulk_deleted`) +- Actor (user ID/email) +- Metadata: `{ item_count: 15, item_ids: [...], outcomes: {...} }` + +**FR-005.10**: Audit log MUST record per-item outcomes: +```json +{ + "item_count": 15, + "succeeded": 13, + "failed": 2, + "skipped": 0, + "failures": [ + {"id": "abc-123", "reason": "Graph API error: 503"}, + {"id": "def-456", "reason": "Policy not found"} + ] +} +``` + +### Progress Tracking + +**FR-005.11**: For queued bulk jobs (≥20 items), system MUST emit progress via: +- `BulkOperationRun` model (status, processed_items updated after each batch) +- Livewire polling on UI (every 3-5 seconds) to fetch updated progress +- Filament notification with progress bar: + - Initial: "Processing... 0/{count}" + - Periodic: "Processing... {done}/{count}" + - Final: "Completed: {succeeded} succeeded, {failed} failed" + +**FR-005.11a**: UI MUST poll `BulkOperationRun` status endpoint (e.g., `/api/bulk-operations/{id}/status`) or use Livewire wire:poll to refresh progress. + +**FR-005.12**: Final notification MUST include link to audit log entry for details. + +**FR-005.13**: If job fails catastrophically (exception), notification MUST show: "Bulk operation failed. Contact support." + +### Error Handling + +**FR-005.14**: System MUST continue processing remaining items if one fails (fail-soft, not fail-fast). + +**FR-005.15**: System MUST collect all failures and report them in final notification + audit log. + +**FR-005.16**: If >50% of items fail, system MUST: +- Abort processing remaining items (status = `aborted`) +- Final notification: "Bulk operation aborted: {failed}/{total} failures exceeded threshold" +- Admin can manually trigger "Retry Failed Items" from BulkOperationRun detail view (future enhancement) + +--- + +## Bulk Actions by Resource + +### Policies Resource + +| Action | Priority | Destructive | Scope | Threshold for Queue | Type-to-Confirm | +|--------|----------|-------------|-------|---------------------|-----------------| +| Delete (local) | P1 | Yes (local only) | TenantPilot DB | ≥20 | ≥20 | +| Export to Backup | P1 | No | TenantPilot DB | ≥20 | No | +| Force Delete | P3 | Yes (local) | TenantPilot DB | ≥10 | Always | +| Restore (untrash) | P3 | No | TenantPilot DB | ≥50 | No | +| Sync (re-fetch) | P4 | No | Graph read | ≥50 | No | + +**FR-005.17**: Bulk Delete for Policies MUST set `ignored_at` timestamp (prevents re-sync) + optionally `deleted_at` (soft delete). Does NOT call Graph DELETE. + +**FR-005.17a**: Sync Job MUST skip policies where `ignored_at IS NOT NULL`. + +**FR-005.18**: Bulk Export to Backup MUST prompt for: +- Backup Set name (auto-generated default: "Bulk Export {date}") +- "Include Assignments" checkbox (if Feature 004 implemented) + +**FR-005.19**: Bulk Sync MUST queue a SyncPoliciesJob for each selected policy. + +### Policy Versions Resource + +| Action | Priority | Destructive | Threshold for Queue | Type-to-Confirm | +|--------|----------|-------------|---------------------|-----------------| +| Delete | P2 | Yes | ≥20 | ≥20 | +| Export to Backup | P3 | No | ≥20 | No | + +**FR-005.20**: Bulk Delete for Policy Versions MUST: +- Check eligibility: `is_current = false` AND `created_at < NOW() - 90 days` AND NOT referenced +- Referenced = exists in `backup_items.policy_version_id` OR `restore_runs.metadata` OR critical audit logs +- Hard-delete eligible versions +- Skip ineligible with reason: "Referenced", "Too recent", "Current version" + +**FR-005.21**: System MUST require `policy_versions.prune` permission (separate from `policy_versions.delete`). + +### Backup Sets Resource + +| Action | Priority | Destructive | Threshold for Queue | Type-to-Confirm | +|--------|----------|-------------|---------------------|-----------------| +| Delete | P2 | Yes | ≥10 | ≥10 | +| Archive (flag) | P3 | No | N/A | No | + +**FR-005.22**: Bulk Delete for Backup Sets MUST cascade-delete related Backup Items. + +**FR-005.23**: Bulk Archive MUST set `archived_at` timestamp (soft flag, keeps data). + +### Restore Runs Resource + +| Action | Priority | Destructive | Threshold for Queue | Type-to-Confirm | +|--------|----------|-------------|---------------------|-----------------| +| Delete | P2 | Yes | ≥20 | ≥20 | +| Rerun | P3 | No | N/A | No | +| Cancel (abort) | P3 | No | N/A | No | + +**FR-005.24**: Bulk Delete for Restore Runs MUST soft-delete. + +**FR-005.25**: Bulk Delete MUST skip runs with status `running` (show warning in results). + +**FR-005.26**: Bulk Rerun (if T156 implemented) MUST create new RestoreRun for each selected run. + +--- + +## Non-Functional Requirements + +**NFR-005.1**: Bulk operations MUST handle up to 500 items per operation without timeout. + +**NFR-005.2**: Queue jobs MUST process items in batches of 10-20 (configurable) to avoid memory issues. + +**NFR-005.3**: Progress notifications MUST update at least every 10 seconds (avoid spamming). + +**NFR-005.4**: UI MUST remain responsive during bulk operations (no blocking spinner). + +**NFR-005.5**: Bulk operations MUST respect tenant isolation (only act on current tenant's data). + +--- + +## Technical Implementation + +### Filament Bulk Actions Setup + +```php +// Example: PolicyResource.php +public static function table(Table $table): Table +{ + return $table + ->columns([...]) + ->bulkActions([ + Tables\Actions\BulkActionGroup::make([ + Tables\Actions\DeleteBulkAction::make() + ->requiresConfirmation() + ->modalHeading(fn (Collection $records) => "Delete {$records->count()} policies?") + ->modalDescription('This moves them to trash.') + ->action(fn (Collection $records) => + BulkPolicyDeleteJob::dispatch($records->pluck('id')) + ), + + Tables\Actions\BulkAction::make('export_to_backup') + ->label('Export to Backup') + ->icon('heroicon-o-arrow-down-tray') + ->form([ + Forms\Components\TextInput::make('backup_name') + ->default('Bulk Export ' . now()->format('Y-m-d')), + Forms\Components\Checkbox::make('include_assignments') + ->label('Include Assignments & Scope Tags'), + ]) + ->action(fn (Collection $records, array $data) => + BulkPolicyExportJob::dispatch($records->pluck('id'), $data) + ), + ]), + ]); +} +``` + +### Queue Job Structure + +```php +// app/Jobs/BulkPolicyDeleteJob.php +class BulkPolicyDeleteJob implements ShouldQueue +{ + use Dispatchable, InteractsWithQueue, Queueable, SerializesModels; + + public function __construct( + public array $policyIds, // array, NOT Collection (serialization) + public int $tenantId, // explicit tenant isolation + public int $actorId, // user ID, not just email + public int $bulkOperationRunId // FK to bulk_operation_runs table + ) {} + + public function handle( + AuditLogger $audit, + PolicyRepository $policies + ): void { + $run = BulkOperationRun::find($this->bulkOperationRunId); + $run->update(['status' => 'running']); + + $results = ['succeeded' => 0, 'failed' => 0, 'skipped' => 0, 'failures' => []]; + + // Process in chunks for memory efficiency + collect($this->policyIds)->chunk(10)->each(function ($chunk) use (&$results, $policies, $run) { + foreach ($chunk as $id) { + try { + $policies->markIgnored($id); // set ignored_at + $results['succeeded']++; + } catch (\Exception $e) { + $results['failed']++; + $results['failures'][] = ['id' => $id, 'reason' => $e->getMessage()]; + } + } + + // Update progress after each chunk + $run->update([ + 'processed_items' => $results['succeeded'] + $results['failed'], + 'succeeded' => $results['succeeded'], + 'failed' => $results['failed'], + 'failures' => $results['failures'], + ]); + + // Circuit breaker: abort if >50% failed + if ($results['failed'] > count($this->policyIds) * 0.5) { + $run->update(['status' => 'aborted']); + throw new \Exception('Bulk operation aborted: >50% failure rate'); + } + }); + + $auditLogId = $audit->log('policies.bulk_deleted_local', [ + 'item_count' => count($this->policyIds), + 'outcomes' => $results, + 'bulk_operation_run_id' => $this->bulkOperationRunId, + ]); + + $run->update(['status' => 'completed', 'audit_log_id' => $auditLogId]); + } +} +``` + +### Type-to-Confirm Modal + +```php +Tables\Actions\DeleteBulkAction::make() + ->requiresConfirmation() + ->modalHeading(fn (Collection $records) => + $records->count() >= 20 + ? "⚠️ Delete {$records->count()} policies?" + : "Delete {$records->count()} policies?" + ) + ->form(fn (Collection $records) => + $records->count() >= 20 + ? [ + Forms\Components\TextInput::make('confirm_delete') + ->label('Type DELETE to confirm') + ->rule('in:DELETE') + ->required() + ->helperText('This action cannot be undone.') + ] + : [] + ) +``` + +--- + +## UI/UX Patterns + +### Bulk Action Menu + +``` +┌────────────────────────────────────────────┐ +│ ☑ Select All (50 items) │ +│ │ +│ 15 selected │ +│ [Delete] [Export to Backup] [More ▾] │ +└────────────────────────────────────────────┘ +``` + +### Confirmation Dialog (≥20 items) + +``` +⚠️ Delete 25 policies? + +This moves them to trash. You can restore them later. + +Type DELETE to confirm: +[________________] + +[Cancel] [Confirm] (disabled until typed) +``` + +### Progress Notification + +``` +🔄 Deleting policies... + ████████████░░░░░░░░ 45 / 100 + +[View Details] +``` + +### Final Notification + +``` +✅ Deleted 98 policies + +2 items failed (click for details) + +[View Audit Log] [Dismiss] +``` + +--- + +## Testing Strategy + +### Unit Tests +- `BulkPolicyDeleteJobTest`: Mock policy repo, test outcomes +- `BulkActionPermissionTest`: Verify permission checks +- `ConfirmationDialogTest`: Test type-to-confirm logic + +### Feature Tests +- `BulkDeletePoliciesTest`: E2E flow (select → confirm → verify soft delete) +- `BulkExportToBackupTest`: E2E export with job queue +- `BulkProgressNotificationTest`: Verify progress events emitted + +### Load Tests +- 500 items bulk delete (should complete in <5 minutes) +- 1000 items bulk export (queue + batch processing) + +### Manual QA +- Select 30 policies → bulk delete → verify trash +- Export 50 policies → verify backup set created +- Test type-to-confirm with correct/incorrect input +- Force job failure → verify error handling + +--- + +## Rollout Plan + +### Phase 1: Foundation (P1 Actions) +- Policies: Bulk Delete, Bulk Export +- Confirmation dialogs + type-to-confirm +- **Duration**: ~8-12 hours + +### Phase 2: Queue + Progress (P1 Features) +- Queue jobs for ≥20 items +- Progress notifications +- Audit logging +- **Duration**: ~8-10 hours + +### Phase 3: Additional Resources (P2 Actions) +- Policy Versions: Bulk Delete +- Restore Runs: Bulk Delete +- Backup Sets: Bulk Delete +- **Duration**: ~6-8 hours + +### Phase 4: Advanced Actions (P3 Optional) +- Bulk Force Delete +- Bulk Restore (untrash) +- Bulk Rerun (depends on T156) +- **Duration**: ~4-6 hours per action + +--- + +## Dependencies +- Laravel Queue (✅ configured) +- Filament Bulk Actions (✅ built-in) +- Feature 001: Audit Logger (✅ complete) + +## Risks & Mitigations + +| Risk | Mitigation | +|------|------------| +| Large batches cause timeout | Queue jobs + chunked processing (10-20 items/batch) + Bus::batch() | +| User accidentally deletes 500 items | Type-to-confirm for ≥20 items + `ignored_at` flag (restorable) | +| Job fails mid-process | Fail-soft, log failures in `bulk_operation_runs`, abort if >50% fail | +| UI becomes unresponsive | Async jobs + Livewire polling for progress | +| Policy Versions deleted while referenced | Eligibility check: not referenced in backups/restores/audits | +| Sync re-adds "deleted" policies | `ignored_at` flag prevents re-sync | +| Progress notifications don't update | `BulkOperationRun` model + polling required (not automatic Filament feature) | + +--- + +## Success Criteria + +1. ✅ Bulk delete 100 policies in <2 minutes (queued) +2. ✅ Type-to-confirm prevents accidental deletes +3. ✅ Progress notifications update every 5-10s +4. ✅ Audit log captures per-item outcomes +5. ✅ 95%+ success rate for bulk operations +6. ✅ Tests cover all P1/P2 actions + +--- + +## Open Questions +1. Should we add bulk "Tag" (apply labels/categories)? +2. Bulk "Clone" for policies (create duplicates)? +3. Max items per bulk operation (hard limit)? +4. Retry failed items in bulk operation? + +--- + +**Status**: Draft for Review +**Created**: 2025-12-22 +**Author**: AI + Ahmed +**Next Steps**: Review → Plan → Tasks