TenantAtlas/specs/005-bulk-operations/spec.md

Feature 005: Bulk Operations for Resource Management

Overview

Enable efficient bulk operations across TenantPilot's main resources (Policies, Policy Versions, Backup Sets, Restore Runs) to improve admin productivity and reduce repetitive actions.

Problem Statement

Currently, admins must perform actions one-by-one on individual resources:

• Deleting 20 old Policy Versions = 20 clicks + confirmations
• Exporting 50 Policies to a Backup = 50 manual selections
• Cleaning up 30 failed Restore Runs = 30 delete actions

This is tedious, error-prone, and time-consuming.

With bulk operations:

• Select multiple items → single action → confirm → done
• Clear audit trail (one bulk action = one audit event + per-item outcomes)
• Progress notifications for long-running operations
• Consistent UX across all resources

Goals

• Primary: Implement bulk delete, bulk export, bulk restore (soft delete) for main resources
• Secondary: Safety gates (confirmation dialogs, type-to-confirm for destructive ops)
• Tertiary: Queue-based processing for large batches with progress tracking
• Non-Goal: Bulk edit/update (too complex, deferred to future feature)

---

User Stories

User Story 1 - Bulk Delete Policies (Priority: P1)

As an admin, I want to soft-delete multiple policies locally in TenantPilot at once, so I can clean up outdated or test policies efficiently.

Important: This action marks policies as deleted locally, does NOT delete them in Intune. Policies are flagged as ignored_at to prevent re-sync.

Acceptance Criteria:

1. Given I select 15 policies in the Policies table, When I click "Delete (Local)" in the bulk actions menu, Then a confirmation dialog appears: "Delete 15 policies locally? They will be hidden from listings and ignored in sync."

2. Given I confirm the bulk delete, When the operation completes, Then:

   • All 15 policies are flagged (ignored_at timestamp set, optionally deleted_at)
   • A success notification shows: "Deleted 15 policies locally"
   • An audit log entry policies.bulk_deleted_local is created with policy IDs
   • Policies remain in Intune (unchanged)

3. Given I bulk-delete 50 policies, When the operation runs, Then it processes asynchronously via queue (job) with progress notification

4. Given I lack policies.delete permission, When I try to bulk-delete, Then the bulk action is disabled/hidden (same permission model as single delete)

---

User Story 2 - Bulk Export Policies to Backup (Priority: P1)

As an admin, I want to export multiple policies to a new Backup Set in one action, so I can quickly snapshot a subset of policies.

Acceptance Criteria:

1. Given I select 25 policies, When I click "Export to Backup", Then a dialog prompts: "Backup Set Name" + "Include Assignments?" checkbox

2. Given I confirm the export, When the backup job runs, Then:

   • A new Backup Set is created
   • 25 Backup Items are captured (one per policy)
   • Progress notification: "Backing up... 10/25"
   • Final notification: "Backup Set 'Production Snapshot' created (25 items)"

3. Given 3 of 25 policies fail to backup (Graph error), When the job completes, Then:

   • 22 items succeed, 3 fail
   • Notification: "Backup completed: 22 succeeded, 3 failed"
   • Audit log records per-item outcomes

---

User Story 3 - Bulk Delete Policy Versions (Priority: P2)

As an admin, I want to bulk-delete old policy versions to free up database space, respecting retention policies.

Important: Policy Versions are immutable snapshots. Deletion only allowed if version is NOT referenced (no active Backup Items, Restore Runs, or audit trails) and meets retention threshold (e.g., >90 days old).

Acceptance Criteria:

1. Given I select 30 policy versions older than 90 days, When I click "Delete", Then confirmation dialog: "Delete 30 policy versions? This is permanent and cannot be undone."

2. Given I confirm, When the operation completes, Then:

   • System checks each version: is_current=false + not referenced + age >90 days
   • Eligible versions are hard-deleted
   • Ineligible versions are skipped with reason (e.g., "Referenced by Backup Set ID 5")
   • Success notification: "Deleted 28 policy versions (2 skipped)"
   • Audit log: policy_versions.bulk_pruned with version IDs + skip reasons

3. Given I lack policy_versions.prune permission, When I try to bulk-delete, Then the bulk action is hidden

---

User Story 4 - Bulk Delete Restore Runs (Priority: P2)

As an admin, I want to bulk-delete completed or failed Restore Runs to declutter the history.

Acceptance Criteria:

1. Given I select 20 restore runs (status: completed/failed), When I click "Delete", Then confirmation: "Delete 20 restore runs? Historical data will be removed."

2. Given I confirm, When the operation completes, Then:

   • 20 restore runs are soft-deleted
   • Notification: "Deleted 20 restore runs"
   • Audit log: restore_runs.bulk_deleted

3. Given I select restore runs with mixed statuses (running + completed), When I attempt bulk delete, Then only completed/failed runs are deleted (running ones skipped with warning)

---

User Story 5 - Bulk Delete with Type-to-Confirm (Priority: P1)

As an admin, I want extra confirmation for large destructive operations, so I don't accidentally delete important data.

Acceptance Criteria:

1. Given I bulk-delete ≥20 items, When the confirmation dialog appears, Then I must type "DELETE" in a text field to enable the confirm button

2. Given I type an incorrect word (e.g., "delete" lowercase), When I try to confirm, Then the button remains disabled with error: "Type DELETE to confirm"

3. Given I type "DELETE" correctly, When I click confirm, Then the bulk operation proceeds

---

User Story 6 - Bulk Operation Progress Tracking (Priority: P2)

As an admin, I want to see real-time progress for bulk operations, so I know the system is working.

Acceptance Criteria:

1. Given I bulk-delete 100 policies, When the job starts, Then a Filament notification shows: "Deleting policies... 0/100"

2. Given the job processes items, When progress updates, Then the notification updates every 5 seconds: "Deleting... 45/100"

3. Given the job completes, When all items are processed, Then:

   • Final notification: "Deleted 98 policies (2 failed)"
   • Clickable link: "View details" → opens audit log entry

---

Functional Requirements

General Bulk Operations

FR-005.1: System MUST provide bulk action checkboxes on table rows for:

• Policies
• Policy Versions
• Backup Sets
• Restore Runs

FR-005.2: Bulk actions menu MUST appear when ≥1 item is selected, showing:

• Action name (e.g., "Delete")
• Count badge (e.g., "3 selected")
• Disabled state if user lacks permission

FR-005.3: System MUST enforce same permissions for bulk actions as single actions (e.g., policies.delete for bulk delete).

FR-005.4: Bulk operations processing ≥20 items MUST run via Laravel Queue (async job) using Bus::batch() or chunked processing (batches of 10-20 items).

FR-005.4a: System MUST create a bulk_operation_runs table to track progress:

Schema::create('bulk_operation_runs', function (Blueprint $table) {
    $table->id();
    $table->foreignId('tenant_id')->constrained();
    $table->foreignId('user_id')->constrained();
    $table->string('resource'); // 'policies', 'policy_versions', etc.
    $table->string('action'); // 'delete', 'export', etc.
    $table->string('status'); // 'running', 'completed', 'failed', 'aborted'
    $table->integer('total_items');
    $table->integer('processed_items')->default(0);
    $table->integer('succeeded')->default(0);
    $table->integer('failed')->default(0);
    $table->integer('skipped')->default(0);
    $table->json('item_ids'); // array of IDs
    $table->json('failures')->nullable(); // [{id, reason}, ...]
    $table->foreignId('audit_log_id')->nullable()->constrained();
    $table->timestamps();
});

**FR-005.5**: Bulk operations <20 items MAY run synchronously (immediate feedback).

### Confirmation Dialogs

**FR-005.6**: Confirmation dialog MUST show:
- Action description: "Delete 15 policies?"
- Impact warning: "This moves them to trash." or "This is permanent."
- Item count badge
- Cancel/Confirm buttons

**FR-005.7**: For destructive operations with ≥20 items, dialog MUST require typing "DELETE" (case-sensitive) to enable confirm button.

**FR-005.8**: For non-destructive operations (export, restore), typing confirmation is NOT required.

### Audit Logging

**FR-005.9**: System MUST create one audit log entry per bulk operation with:
- Event type: `{resource}.bulk_{action}` (e.g., `policies.bulk_deleted`)
- Actor (user ID/email)
- Metadata: `{ item_count: 15, item_ids: [...], outcomes: {...} }`

**FR-005.10**: Audit log MUST record per-item outcomes:
```json
{
  "item_count": 15,
  "succeeded": 13,
  "failed": 2,
  "skipped": 0,
  "failures": [
    {"id": "abc-123", "reason": "Graph API error: 503"},
    {"id": "def-456", "reason": "Policy not found"}
  ]
}

Progress Tracking

FR-005.11: For queued bulk jobs (≥20 items), system MUST emit progress via:

• BulkOperationRun model (status, processed_items updated after each batch)
• Livewire polling on UI (every 3-5 seconds) to fetch updated progress
• Filament notification with progress bar:
  • Initial: "Processing... 0/{count}"
  • Periodic: "Processing... {done}/{count}"
  • Final: "Completed: {succeeded} succeeded, {failed} failed"

FR-005.11a: UI MUST poll BulkOperationRun status endpoint (e.g., /api/bulk-operations/{id}/status) or use Livewire wire:poll to refresh progress.

FR-005.12: Final notification MUST include link to audit log entry for details.

FR-005.13: If job fails catastrophically (exception), notification MUST show: "Bulk operation failed. Contact support."

Error Handling

FR-005.14: System MUST continue processing remaining items if one fails (fail-soft, not fail-fast).

FR-005.15: System MUST collect all failures and report them in final notification + audit log.

FR-005.16: If >50% of items fail, system MUST:

• Abort processing remaining items (status = aborted)
• Final notification: "Bulk operation aborted: {failed}/{total} failures exceeded threshold"
• Admin can manually trigger "Retry Failed Items" from BulkOperationRun detail view (future enhancement)

---

Bulk Actions by Resource

Policies Resource

Action	Priority	Destructive	Scope	Threshold for Queue	Type-to-Confirm
Delete (local)	P1	Yes (local only)	TenantPilot DB	≥20	≥20
Export to Backup	P1	No	TenantPilot DB	≥20	No
Force Delete	P3	Yes (local)	TenantPilot DB	≥10	Always
Restore (untrash)	P3	No	TenantPilot DB	≥50	No
Sync (re-fetch)	P4	No	Graph read	≥50	No

FR-005.17: Bulk Delete for Policies MUST set ignored_at timestamp (prevents re-sync) + optionally deleted_at (soft delete). Does NOT call Graph DELETE.

FR-005.17a: Sync Job MUST skip policies where ignored_at IS NOT NULL.

FR-005.18: Bulk Export to Backup MUST prompt for:

• Backup Set name (auto-generated default: "Bulk Export {date}")
• "Include Assignments" checkbox (if Feature 004 implemented)

FR-005.19: Bulk Sync MUST queue a SyncPoliciesJob for each selected policy.

Policy Versions Resource

Action	Priority	Destructive	Threshold for Queue	Type-to-Confirm
Delete	P2	Yes	≥20	≥20
Export to Backup	P3	No	≥20	No

FR-005.20: Bulk Delete for Policy Versions MUST:

• Check eligibility: is_current = false AND created_at < NOW() - 90 days AND NOT referenced
• Referenced = exists in backup_items.policy_version_id OR restore_runs.metadata OR critical audit logs
• Hard-delete eligible versions
• Skip ineligible with reason: "Referenced", "Too recent", "Current version"

FR-005.21: System MUST require policy_versions.prune permission (separate from policy_versions.delete).

Backup Sets Resource

Action	Priority	Destructive	Threshold for Queue	Type-to-Confirm
Delete	P2	Yes	≥10	≥10
Archive (flag)	P3	No	N/A	No

FR-005.22: Bulk Delete for Backup Sets MUST cascade-delete related Backup Items.

FR-005.23: Bulk Archive MUST set archived_at timestamp (soft flag, keeps data).

Restore Runs Resource

Action	Priority	Destructive	Threshold for Queue	Type-to-Confirm
Delete	P2	Yes	≥20	≥20
Rerun	P3	No	N/A	No
Cancel (abort)	P3	No	N/A	No

FR-005.24: Bulk Delete for Restore Runs MUST soft-delete.

FR-005.25: Bulk Delete MUST skip runs with status running (show warning in results).

FR-005.26: Bulk Rerun (if T156 implemented) MUST create new RestoreRun for each selected run.

---

Non-Functional Requirements

NFR-005.1: Bulk operations MUST handle up to 500 items per operation without timeout.

NFR-005.2: Queue jobs MUST process items in batches of 10-20 (configurable) to avoid memory issues.

NFR-005.3: Progress notifications MUST update at least every 10 seconds (avoid spamming).

NFR-005.4: UI MUST remain responsive during bulk operations (no blocking spinner).

NFR-005.5: Bulk operations MUST respect tenant isolation (only act on current tenant's data).

---

Technical Implementation

Filament Bulk Actions Setup

// Example: PolicyResource.php
public static function table(Table $table): Table
{
    return $table
        ->columns([...])
        ->bulkActions([
            Tables\Actions\BulkActionGroup::make([
                Tables\Actions\DeleteBulkAction::make()
                    ->requiresConfirmation()
                    ->modalHeading(fn (Collection $records) => "Delete {$records->count()} policies?")
                    ->modalDescription('This moves them to trash.')
                    ->action(fn (Collection $records) => 
                        BulkPolicyDeleteJob::dispatch($records->pluck('id'))
                    ),
                    
                Tables\Actions\BulkAction::make('export_to_backup')
                    ->label('Export to Backup')
                    ->icon('heroicon-o-arrow-down-tray')
                    ->form([
                        Forms\Components\TextInput::make('backup_name')
                            ->default('Bulk Export ' . now()->format('Y-m-d')),
                        Forms\Components\Checkbox::make('include_assignments')
                            ->label('Include Assignments & Scope Tags'),
                    ])
                    ->action(fn (Collection $records, array $data) => 
                        BulkPolicyExportJob::dispatch($records->pluck('id'), $data)
                    ),
            ]),
        ]);
}

Queue Job Structure

// app/Jobs/BulkPolicyDeleteJob.php
class BulkPolicyDeleteJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public function __construct(
        public array $policyIds,      // array, NOT Collection (serialization)
        public int $tenantId,          // explicit tenant isolation
        public int $actorId,           // user ID, not just email
        public int $bulkOperationRunId // FK to bulk_operation_runs table
    ) {}

    public function handle(
        AuditLogger $audit,
        PolicyRepository $policies
    ): void {
        $run = BulkOperationRun::find($this->bulkOperationRunId);
        $run->update(['status' => 'running']);
        
        $results = ['succeeded' => 0, 'failed' => 0, 'skipped' => 0, 'failures' => []];
        
        // Process in chunks for memory efficiency
        collect($this->policyIds)->chunk(10)->each(function ($chunk) use (&$results, $policies, $run) {
            foreach ($chunk as $id) {
                try {
                    $policies->markIgnored($id); // set ignored_at
                    $results['succeeded']++;
                } catch (\Exception $e) {
                    $results['failed']++;
                    $results['failures'][] = ['id' => $id, 'reason' => $e->getMessage()];
                }
            }
            
            // Update progress after each chunk
            $run->update([
                'processed_items' => $results['succeeded'] + $results['failed'],
                'succeeded' => $results['succeeded'],
                'failed' => $results['failed'],
                'failures' => $results['failures'],
            ]);
            
            // Circuit breaker: abort if >50% failed
            if ($results['failed'] > count($this->policyIds) * 0.5) {
                $run->update(['status' => 'aborted']);
                throw new \Exception('Bulk operation aborted: >50% failure rate');
            }
        });
        
        $auditLogId = $audit->log('policies.bulk_deleted_local', [
            'item_count' => count($this->policyIds),
            'outcomes' => $results,
            'bulk_operation_run_id' => $this->bulkOperationRunId,
        ]);
        
        $run->update(['status' => 'completed', 'audit_log_id' => $auditLogId]);
    }
}

Type-to-Confirm Modal

Tables\Actions\DeleteBulkAction::make()
    ->requiresConfirmation()
    ->modalHeading(fn (Collection $records) => 
        $records->count() >= 20 
            ? "⚠️ Delete {$records->count()} policies?" 
            : "Delete {$records->count()} policies?"
    )
    ->form(fn (Collection $records) => 
        $records->count() >= 20 
            ? [
                Forms\Components\TextInput::make('confirm_delete')
                    ->label('Type DELETE to confirm')
                    ->rule('in:DELETE')
                    ->required()
                    ->helperText('This action cannot be undone.')
              ]
            : []
    )

---

UI/UX Patterns

Bulk Action Menu

┌────────────────────────────────────────────┐
│ ☑ Select All (50 items)                   │
│                                            │
│ 15 selected                                │
│ [Delete] [Export to Backup] [More ▾]       │
└────────────────────────────────────────────┘

Confirmation Dialog (≥20 items)

⚠️ Delete 25 policies?

This moves them to trash. You can restore them later.

Type DELETE to confirm:
[________________]

[Cancel]  [Confirm] (disabled until typed)

Progress Notification

🔄 Deleting policies...
   ████████████░░░░░░░░  45 / 100

[View Details]

Final Notification

✅ Deleted 98 policies

2 items failed (click for details)

[View Audit Log]  [Dismiss]

---

Testing Strategy

Unit Tests

• BulkPolicyDeleteJobTest: Mock policy repo, test outcomes
• BulkActionPermissionTest: Verify permission checks
• ConfirmationDialogTest: Test type-to-confirm logic

Feature Tests

• BulkDeletePoliciesTest: E2E flow (select → confirm → verify soft delete)
• BulkExportToBackupTest: E2E export with job queue
• BulkProgressNotificationTest: Verify progress events emitted

Load Tests

• 500 items bulk delete (should complete in <5 minutes)
• 1000 items bulk export (queue + batch processing)

Manual QA

• Select 30 policies → bulk delete → verify trash
• Export 50 policies → verify backup set created
• Test type-to-confirm with correct/incorrect input
• Force job failure → verify error handling

---

Rollout Plan

Phase 1: Foundation (P1 Actions)

• Policies: Bulk Delete, Bulk Export
• Confirmation dialogs + type-to-confirm
• Duration: ~8-12 hours

Phase 2: Queue + Progress (P1 Features)

• Queue jobs for ≥20 items
• Progress notifications
• Audit logging
• Duration: ~8-10 hours

Phase 3: Additional Resources (P2 Actions)

• Policy Versions: Bulk Delete
• Restore Runs: Bulk Delete
• Backup Sets: Bulk Delete
• Duration: ~6-8 hours

Phase 4: Advanced Actions (P3 Optional)

• Bulk Force Delete
• Bulk Restore (untrash)
• Bulk Rerun (depends on T156)
• Duration: ~4-6 hours per action

---

Dependencies

• Laravel Queue (✅ configured)
• Filament Bulk Actions (✅ built-in)
• Feature 001: Audit Logger (✅ complete)

Risks & Mitigations

Risk	Mitigation
Large batches cause timeout	Queue jobs + chunked processing (10-20 items/batch) + Bus::batch()
User accidentally deletes 500 items	Type-to-confirm for ≥20 items + ignored_at flag (restorable)
Job fails mid-process	Fail-soft, log failures in bulk_operation_runs, abort if >50% fail
UI becomes unresponsive	Async jobs + Livewire polling for progress
Policy Versions deleted while referenced	Eligibility check: not referenced in backups/restores/audits
Sync re-adds "deleted" policies	ignored_at flag prevents re-sync
Progress notifications don't update	BulkOperationRun model + polling required (not automatic Filament feature)

---

Success Criteria

1. ✅ Bulk delete 100 policies in <2 minutes (queued)
2. ✅ Type-to-confirm prevents accidental deletes
3. ✅ Progress notifications update every 5-10s
4. ✅ Audit log captures per-item outcomes
5. ✅ 95%+ success rate for bulk operations
6. ✅ Tests cover all P1/P2 actions

---

Open Questions

1. Should we add bulk "Tag" (apply labels/categories)?
2. Bulk "Clone" for policies (create duplicates)?
3. Max items per bulk operation (hard limit)?
4. Retry failed items in bulk operation?

---

Status: Draft for Review
Created: 2025-12-22
Author: AI + Ahmed
Next Steps: Review → Plan → Tasks