From 673fbd6b22be153f3381999195053577a6bc2079 Mon Sep 17 00:00:00 2001 From: Ahmed Darrazi Date: Mon, 22 Dec 2025 01:27:42 +0100 Subject: [PATCH] spec(005): Add comprehensive implementation plan for Bulk Operations Adds: - plan.md: Technical context, constitution check, phases - research.md: 7 research decisions (progress tracking, chunking, type-to-confirm) - data-model.md: BulkOperationRun model, schema changes, query patterns - quickstart.md: Developer onboarding, testing workflows, debugging Key Decisions: - BulkOperationRun model + Livewire polling for progress - collect()->chunk(10) for memory-efficient processing - Filament form + validation for type-to-confirm - ignored_at flag to prevent sync re-adding deleted policies - Eligibility scopes for safe Policy Version pruning Estimated: 26-34 hours (3 phases for P1/P2 features) Next: /speckit.tasks to generate task breakdown --- .github/agents/copilot-instructions.md | 29 + specs/005-bulk-operations/data-model.md | 711 ++++++++++++++++++++++++ specs/005-bulk-operations/plan.md | 263 +++++++++ specs/005-bulk-operations/quickstart.md | 425 ++++++++++++++ specs/005-bulk-operations/research.md | 547 ++++++++++++++++++ 5 files changed, 1975 insertions(+) create mode 100644 .github/agents/copilot-instructions.md create mode 100644 specs/005-bulk-operations/data-model.md create mode 100644 specs/005-bulk-operations/plan.md create mode 100644 specs/005-bulk-operations/quickstart.md create mode 100644 specs/005-bulk-operations/research.md diff --git a/.github/agents/copilot-instructions.md b/.github/agents/copilot-instructions.md new file mode 100644 index 0000000..62118e0 --- /dev/null +++ b/.github/agents/copilot-instructions.md @@ -0,0 +1,29 @@ +# TenantAtlas Development Guidelines + +Auto-generated from all feature plans. Last updated: 2025-12-22 + +## Active Technologies + +- PHP 8.4.15 (feat/005-bulk-operations) + +## Project Structure + +```text +src/ +tests/ +``` + +## Commands + +# Add commands for PHP 8.4.15 + +## Code Style + +PHP 8.4.15: Follow standard conventions + +## Recent Changes + +- feat/005-bulk-operations: Added PHP 8.4.15 + + + diff --git a/specs/005-bulk-operations/data-model.md b/specs/005-bulk-operations/data-model.md new file mode 100644 index 0000000..47003c1 --- /dev/null +++ b/specs/005-bulk-operations/data-model.md @@ -0,0 +1,711 @@ +# Data Model: Feature 005 - Bulk Operations + +**Feature**: Bulk Operations for Resource Management +**Date**: 2025-12-22 + +--- + +## Overview + +This document describes the data model for bulk operations, including new entities, schema changes, relationships, and query patterns. + +--- + +## Entity Diagram + +``` +┌─────────────────────────────────────────────────────────────┐ +│ BulkOperationRun │ +├─────────────────────────────────────────────────────────────┤ +│ id: bigint PK │ +│ tenant_id: bigint FK → Tenant │ +│ user_id: bigint FK → User │ +│ resource: string (policies, policy_versions, etc.) │ +│ action: string (delete, export, prune, etc.) │ +│ status: enum (pending, running, completed, failed, aborted) │ +│ total_items: int │ +│ processed_items: int │ +│ succeeded: int │ +│ failed: int │ +│ skipped: int │ +│ item_ids: jsonb (array of IDs) │ +│ failures: jsonb (array of {id, reason}) │ +│ audit_log_id: bigint FK → AuditLog (nullable) │ +│ created_at: timestamp │ +│ updated_at: timestamp │ +└─────────────────────────────────────────────────────────────┘ + │ │ │ + │ │ └─────────────┐ + │ │ │ + ▼ ▼ ▼ +┌──────────────┐ ┌──────────────┐ ┌──────────────┐ +│ Tenant │ │ User │ │ AuditLog │ +└──────────────┘ └──────────────┘ └──────────────┘ + + +┌─────────────────────────────────────────┐ +│ Policy (Extended) │ +├─────────────────────────────────────────┤ +│ id: bigint PK │ +│ tenant_id: bigint FK │ +│ graph_id: string │ +│ name: string │ +│ ... (existing columns) │ +│ deleted_at: timestamp (nullable) │ +│ ignored_at: timestamp (nullable) ← NEW │ +└─────────────────────────────────────────┘ + + +┌─────────────────────────────────────────┐ +│ PolicyVersion (Extended) │ +├─────────────────────────────────────────┤ +│ id: bigint PK │ +│ policy_id: bigint FK → Policy │ +│ is_current: boolean │ +│ created_at: timestamp │ +│ ... (existing columns) │ +├─────────────────────────────────────────┤ +│ Scope: pruneEligible() ← NEW │ +│ WHERE is_current = false │ +│ AND created_at < NOW() - 90 days │ +│ AND NOT IN (backup_items) │ +│ AND NOT IN (restore_runs.metadata) │ +└─────────────────────────────────────────┘ + + +┌─────────────────────────────────────────┐ +│ RestoreRun (Extended) │ +├─────────────────────────────────────────┤ +│ id: bigint PK │ +│ status: enum (pending, running, ...) │ +│ ... (existing columns) │ +├─────────────────────────────────────────┤ +│ Scope: deletable() ← NEW │ +│ WHERE status IN (completed, failed) │ +└─────────────────────────────────────────┘ +``` + +--- + +## Database Schema + +### New Table: bulk_operation_runs + +```sql +CREATE TABLE bulk_operation_runs ( + id BIGSERIAL PRIMARY KEY, + tenant_id BIGINT NOT NULL REFERENCES tenants(id) ON DELETE CASCADE, + user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE CASCADE, + resource VARCHAR(50) NOT NULL, -- 'policies', 'policy_versions', 'backup_sets', 'restore_runs' + action VARCHAR(50) NOT NULL, -- 'delete', 'export', 'prune', 'archive' + status VARCHAR(20) NOT NULL, -- 'pending', 'running', 'completed', 'failed', 'aborted' + total_items INT NOT NULL, + processed_items INT NOT NULL DEFAULT 0, + succeeded INT NOT NULL DEFAULT 0, + failed INT NOT NULL DEFAULT 0, + skipped INT NOT NULL DEFAULT 0, + item_ids JSONB NOT NULL, -- [1, 2, 3, ...] + failures JSONB, -- [{"id": 1, "reason": "Graph error"}, ...] + audit_log_id BIGINT REFERENCES audit_logs(id) ON DELETE SET NULL, + created_at TIMESTAMP NOT NULL DEFAULT NOW(), + updated_at TIMESTAMP NOT NULL DEFAULT NOW() +); + +CREATE INDEX idx_bulk_operation_runs_tenant_resource_status + ON bulk_operation_runs(tenant_id, resource, status); + +CREATE INDEX idx_bulk_operation_runs_user_created + ON bulk_operation_runs(user_id, created_at DESC); + +CREATE INDEX idx_bulk_operation_runs_status + ON bulk_operation_runs(status) + WHERE status IN ('pending', 'running'); +``` + +**Indexes Rationale**: +- Composite (tenant_id, resource, status): Filter by tenant + resource type + status (UI queries) +- (user_id, created_at): User's recent bulk operations +- Partial index on status: Only index running/pending (99% of queries check these) + +--- + +### Schema Change: policies table + +```sql +ALTER TABLE policies +ADD COLUMN ignored_at TIMESTAMP NULL; + +CREATE INDEX idx_policies_ignored_at + ON policies(ignored_at) + WHERE ignored_at IS NOT NULL; +``` + +**Purpose**: Prevent SyncPoliciesJob from re-importing locally deleted policies. + +**Query Pattern**: +```sql +-- Sync job filters +SELECT * FROM policies WHERE ignored_at IS NULL; + +-- Bulk delete sets +UPDATE policies SET ignored_at = NOW() WHERE id IN (...); +``` + +--- + +### Schema Change: policy_versions table + +**No schema changes needed.** Eligibility scope uses existing columns: +- `is_current` (boolean) +- `created_at` (timestamp) +- Foreign keys checked via relationships + +--- + +## Eloquent Models + +### New Model: BulkOperationRun + +```php + 'array', + 'failures' => 'array', + 'created_at' => 'datetime', + 'updated_at' => 'datetime', + ]; + + // Relationships + + public function tenant(): BelongsTo + { + return $this->belongsTo(Tenant::class); + } + + public function user(): BelongsTo + { + return $this->belongsTo(User::class); + } + + public function auditLog(): BelongsTo + { + return $this->belongsTo(AuditLog::class); + } + + // Status Helpers + + public function isRunning(): bool + { + return $this->status === 'running'; + } + + public function isComplete(): bool + { + return in_array($this->status, ['completed', 'failed', 'aborted']); + } + + public function isPending(): bool + { + return $this->status === 'pending'; + } + + // Progress Helpers + + public function progressPercentage(): int + { + if ($this->total_items === 0) { + return 0; + } + return (int) round(($this->processed_items / $this->total_items) * 100); + } + + public function summaryText(): string + { + return match ($this->status) { + 'pending' => "Pending: {$this->total_items} items", + 'running' => "Processing... {$this->processed_items}/{$this->total_items}", + 'completed' => "Completed: {$this->succeeded} succeeded, {$this->failed} failed, {$this->skipped} skipped", + 'failed' => "Failed: {$this->failed}/{$this->total_items} errors", + 'aborted' => "Aborted: Too many failures ({$this->failed}/{$this->total_items})", + default => "Unknown status", + }; + } + + public function hasFailures(): bool + { + return $this->failed > 0; + } + + // Scopes + + public function scopeForResource($query, string $resource) + { + return $query->where('resource', $resource); + } + + public function scopeForUser($query, int $userId) + { + return $query->where('user_id', $userId); + } + + public function scopeRecent($query) + { + return $query->orderBy('created_at', 'desc'); + } + + public function scopeInProgress($query) + { + return $query->whereIn('status', ['pending', 'running']); + } +} +``` + +--- + +### Extended Model: Policy + +```php +whereNull('ignored_at'); + } + + public function scopeIgnored($query) + { + return $query->whereNotNull('ignored_at'); + } + + // NEW: Methods for bulk operations + + public function markIgnored(): void + { + $this->update(['ignored_at' => now()]); + } + + public function unmarkIgnored(): void + { + $this->update(['ignored_at' => null]); + } + + public function isIgnored(): bool + { + return $this->ignored_at !== null; + } +} +``` + +--- + +### Extended Model: PolicyVersion + +```php +hasMany(BackupItem::class, 'policy_version_id'); + } + + // NEW: Scope for eligibility check + + public function scopePruneEligible($query, int $retentionDays = 90) + { + return $query + ->where('is_current', false) + ->where('created_at', '<', now()->subDays($retentionDays)) + ->whereDoesntHave('backupItems') + ->whereNotIn('id', function ($subquery) { + $subquery->select(DB::raw("CAST(metadata->>'policy_version_id' AS INTEGER)")) + ->from('restore_runs') + ->whereNotNull(DB::raw("metadata->>'policy_version_id'")); + }); + } + + // NEW: Check if version is eligible for pruning + + public function isPruneEligible(int $retentionDays = 90): bool + { + if ($this->is_current) { + return false; + } + + if ($this->created_at->diffInDays(now()) < $retentionDays) { + return false; + } + + if ($this->backupItems()->exists()) { + return false; + } + + $referencedInRestoreRuns = DB::table('restore_runs') + ->whereNotNull(DB::raw("metadata->>'policy_version_id'")) + ->where(DB::raw("CAST(metadata->>'policy_version_id' AS INTEGER)"), $this->id) + ->exists(); + + return !$referencedInRestoreRuns; + } + + // NEW: Get reason why version is not eligible + + public function getIneligibilityReason(int $retentionDays = 90): ?string + { + if ($this->is_current) { + return 'Current version'; + } + + if ($this->created_at->diffInDays(now()) < $retentionDays) { + return "Too recent (< {$retentionDays} days)"; + } + + if ($this->backupItems()->exists()) { + return 'Referenced in backup items'; + } + + $referencedInRestoreRuns = DB::table('restore_runs') + ->whereNotNull(DB::raw("metadata->>'policy_version_id'")) + ->where(DB::raw("CAST(metadata->>'policy_version_id' AS INTEGER)"), $this->id) + ->exists(); + + if ($referencedInRestoreRuns) { + return 'Referenced in restore runs'; + } + + return null; // Eligible + } +} +``` + +--- + +### Extended Model: RestoreRun + +```php +whereIn('status', ['completed', 'failed', 'aborted']); + } + + public function scopeNotDeletable($query) + { + return $query->whereNotIn('status', ['completed', 'failed', 'aborted']); + } + + // NEW: Check if restore run can be deleted + + public function isDeletable(): bool + { + return in_array($this->status, ['completed', 'failed', 'aborted']); + } + + public function getNotDeletableReason(): ?string + { + if ($this->isDeletable()) { + return null; + } + + return match ($this->status) { + 'pending' => 'Restore run is pending', + 'running' => 'Restore run is currently running', + default => "Restore run has status: {$this->status}", + }; + } +} +``` + +--- + +## Query Patterns + +### 1. Create Bulk Operation Run + +```php +$run = BulkOperationRun::create([ + 'tenant_id' => $tenantId, + 'user_id' => $userId, + 'resource' => 'policies', + 'action' => 'delete', + 'status' => 'pending', + 'total_items' => count($policyIds), + 'item_ids' => $policyIds, +]); + +// Dispatch job +BulkPolicyDeleteJob::dispatch($policyIds, $tenantId, $userId, $run->id); + +$run->update(['status' => 'running']); +``` + +### 2. Update Progress (in job) + +```php +$run->update([ + 'processed_items' => $run->processed_items + $chunkSize, + 'succeeded' => $run->succeeded + $successCount, + 'failed' => $run->failed + $failCount, + 'failures' => array_merge($run->failures ?? [], $newFailures), +]); +``` + +### 3. Query Recent Bulk Operations (UI) + +```php +// User's recent operations +$runs = BulkOperationRun::forUser(auth()->id()) + ->recent() + ->limit(10) + ->get(); + +// Tenant's in-progress operations +$inProgress = BulkOperationRun::where('tenant_id', $tenantId) + ->inProgress() + ->get(); +``` + +### 4. Filter Policies (exclude ignored) + +```php +// Sync job +$policies = Policy::where('tenant_id', $tenantId) + ->notIgnored() + ->get(); + +// Bulk delete (mark as ignored) +Policy::whereIn('id', $policyIds) + ->update(['ignored_at' => now()]); +``` + +### 5. Check Policy Version Eligibility + +```php +// Get all eligible versions for tenant +$eligibleVersions = PolicyVersion::whereHas('policy', function ($query) use ($tenantId) { + $query->where('tenant_id', $tenantId); + }) + ->pruneEligible(90) + ->get(); + +// Check single version +$version = PolicyVersion::find($id); +if (!$version->isPruneEligible()) { + $reason = $version->getIneligibilityReason(); + // Skip with reason: "Referenced in backup items" +} +``` + +### 6. Filter Deletable Restore Runs + +```php +// Get deletable runs +$deletableRuns = RestoreRun::where('tenant_id', $tenantId) + ->deletable() + ->get(); + +// Check individual run +$run = RestoreRun::find($id); +if (!$run->isDeletable()) { + $reason = $run->getNotDeletableReason(); + // Skip: "Restore run is currently running" +} +``` + +--- + +## JSONB Structure + +### bulk_operation_runs.item_ids + +```json +[1, 2, 3, 4, 5, ...] +``` + +Simple array of integer IDs. + +### bulk_operation_runs.failures + +```json +[ + { + "id": 123, + "reason": "Graph API error: 503 Service Unavailable" + }, + { + "id": 456, + "reason": "Policy not found" + }, + { + "id": 789, + "reason": "Permission denied" + } +] +``` + +Array of objects with `id` (resource ID) and `reason` (error message). + +### restore_runs.metadata (existing) + +```json +{ + "policy_version_id": 42, + "backup_set_id": 15, + "items_count": 10, + ... +} +``` + +When checking eligibility, query: +```sql +SELECT * FROM restore_runs +WHERE metadata->>'policy_version_id' = '42'; +``` + +--- + +## Migration Files + +### Migration 1: Create bulk_operation_runs table + +```php +id(); + $table->foreignId('tenant_id')->constrained()->cascadeOnDelete(); + $table->foreignId('user_id')->constrained()->cascadeOnDelete(); + $table->string('resource', 50); + $table->string('action', 50); + $table->string('status', 20); + $table->integer('total_items'); + $table->integer('processed_items')->default(0); + $table->integer('succeeded')->default(0); + $table->integer('failed')->default(0); + $table->integer('skipped')->default(0); + $table->json('item_ids'); + $table->json('failures')->nullable(); + $table->foreignId('audit_log_id')->nullable()->constrained()->nullOnDelete(); + $table->timestamps(); + + $table->index(['tenant_id', 'resource', 'status']); + $table->index(['user_id', 'created_at']); + $table->index('status')->where('status', 'in', ['pending', 'running']); + }); + } + + public function down(): void + { + Schema::dropIfExists('bulk_operation_runs'); + } +}; +``` + +### Migration 2: Add ignored_at to policies + +```php +timestamp('ignored_at')->nullable()->after('deleted_at'); + $table->index('ignored_at'); + }); + } + + public function down(): void + { + Schema::table('policies', function (Blueprint $table) { + $table->dropIndex(['ignored_at']); + $table->dropColumn('ignored_at'); + }); + } +}; +``` + +--- + +## Data Retention & Cleanup + +### BulkOperationRun Retention + +Recommended: Keep for 90 days, then archive or delete completed runs. + +```php +// Scheduled command +Artisan::command('bulk-operations:prune', function () { + $deleted = BulkOperationRun::where('created_at', '<', now()->subDays(90)) + ->whereIn('status', ['completed', 'failed', 'aborted']) + ->delete(); + + $this->info("Pruned {$deleted} old bulk operation runs"); +})->daily(); +``` + +### PolicyVersion Retention + +Handled by bulk prune feature (user-initiated, not automatic). + +--- + +**Status**: Data Model Complete +**Next Step**: Generate quickstart.md diff --git a/specs/005-bulk-operations/plan.md b/specs/005-bulk-operations/plan.md new file mode 100644 index 0000000..0a79cf8 --- /dev/null +++ b/specs/005-bulk-operations/plan.md @@ -0,0 +1,263 @@ +# Implementation Plan: Feature 005 - Bulk Operations + +**Branch**: `feat/005-bulk-operations` | **Date**: 2025-12-22 | **Spec**: [spec.md](./spec.md) + +## Summary + +Enable efficient bulk operations (delete, export, prune) across TenantPilot's main resources (Policies, Policy Versions, Backup Sets, Restore Runs) with safety gates, progress tracking, and comprehensive audit logging. Technical approach: Filament bulk actions + Laravel Queue jobs with chunked processing + BulkOperationRun tracking model + Livewire polling for progress updates. + +## Technical Context + +**Language/Version**: PHP 8.4.15 +**Framework**: Laravel 12 +**Primary Dependencies**: +- Filament v4 (admin panel + bulk actions) +- Livewire v3 (reactive UI + polling) +- Laravel Queue (async job processing) +- PostgreSQL (JSONB for tracking) + +**Storage**: PostgreSQL with JSONB fields for: +- `bulk_operation_runs.item_ids` (array of resource IDs) +- `bulk_operation_runs.failures` (per-item error details) +- Existing audit logs (metadata column) + +**Testing**: Pest v4 (unit, feature, browser tests) +**Target Platform**: Web (Dokploy deployment) +**Project Type**: Web application (Filament admin panel) + +**Performance Goals**: +- Process 100 items in <2 minutes (queued) +- Handle up to 500 items per operation without timeout +- Progress notifications update every 5-10 seconds + +**Constraints**: +- Queue jobs MUST process in chunks of 10-20 items (memory efficiency) +- Progress tracking requires explicit polling (not automatic in Filament) +- Type-to-confirm required for ≥20 destructive items +- Tenant isolation enforced at job level + +**Scale/Scope**: +- 4 primary resources (Policies, PolicyVersions, BackupSets, RestoreRuns) +- 8-12 bulk actions (P1/P2 priority) +- Estimated 26-34 hours implementation (3 phases for P1/P2) + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +**Note**: Project constitution is template-only (not populated). Using Laravel/TenantPilot conventions instead. + +### Architecture Principles + +✅ **Library-First**: N/A (feature extends existing app, no new libraries) +✅ **Test-First**: TDD enforced - Pest tests required before implementation +✅ **Simplicity**: Uses existing patterns (Jobs, Filament bulk actions, Livewire polling) +✅ **Sail-First**: Local development uses Laravel Sail (Docker) +✅ **Dokploy Deployment**: Production/staging via Dokploy (VPS containers) + +### Laravel Conventions + +✅ **PSR-12**: Code formatting enforced via Laravel Pint +✅ **Eloquent-First**: No raw DB queries, use Model::query() patterns +✅ **Permission Gates**: Leverage existing RBAC (Feature 001) +✅ **Queue Jobs**: Use ShouldQueue interface, chunked processing +✅ **Audit Logging**: Extend existing AuditLog model/service + +### Safety Requirements + +✅ **Tenant Isolation**: Job constructor accepts explicit `tenantId` +✅ **Audit Trail**: One audit log entry per bulk operation + per-item outcomes +✅ **Confirmation**: Type-to-confirm for ≥20 destructive items +✅ **Fail-Soft**: Continue processing on individual failures, abort if >50% fail +✅ **Immutability**: Policy Versions check eligibility before prune (referenced, current, age) + +### Gates + +🔒 **GATE-01**: Bulk operations MUST use existing permission model (policies.delete, etc.) +🔒 **GATE-02**: Progress tracking MUST use BulkOperationRun model (not fire-and-forget) +🔒 **GATE-03**: Type-to-confirm MUST be case-sensitive "DELETE" for ≥20 items +🔒 **GATE-04**: Policies bulk delete = local only (ignored_at flag, NO Graph DELETE) + +## Project Structure + +### Documentation (this feature) + +```text +specs/005-bulk-operations/ +├── plan.md # This file +├── research.md # Phase 0 output (see below) +├── data-model.md # Phase 1 output (see below) +├── quickstart.md # Phase 1 output (see below) +└── tasks.md # Phase 2 output (/speckit.tasks command - NOT YET CREATED) +``` + +### Source Code (repository root) + +```text +app/ +├── Models/ +│ ├── BulkOperationRun.php # NEW: Tracks progress/outcomes +│ ├── Policy.php # EXTEND: Add markIgnored() scope +│ ├── PolicyVersion.php # EXTEND: Add pruneEligible() scope +│ ├── BackupSet.php # EXTEND: Cascade delete logic +│ └── RestoreRun.php # EXTEND: Skip running status +│ +├── Jobs/ +│ ├── BulkPolicyDeleteJob.php # NEW: Async bulk delete (local) +│ ├── BulkPolicyExportJob.php # NEW: Export to backup set +│ ├── BulkPolicyVersionPruneJob.php # NEW: Prune old versions +│ ├── BulkBackupSetDeleteJob.php # NEW: Delete backup sets +│ └── BulkRestoreRunDeleteJob.php # NEW: Delete restore runs +│ +├── Services/ +│ ├── BulkOperationService.php # NEW: Orchestrates bulk ops + tracking +│ └── Audit/ +│ └── AuditLogger.php # EXTEND: Add bulk operation events +│ +├── Filament/ +│ └── Resources/ +│ ├── PolicyResource.php # EXTEND: Add bulk actions +│ ├── PolicyVersionResource.php # EXTEND: Add bulk prune +│ ├── BackupSetResource.php # EXTEND: Add bulk delete +│ └── RestoreRunResource.php # EXTEND: Add bulk delete +│ +└── Livewire/ + └── BulkOperationProgress.php # NEW: Progress polling component + +database/ +└── migrations/ + └── YYYY_MM_DD_create_bulk_operation_runs_table.php # NEW + +tests/ +├── Unit/ +│ ├── BulkPolicyDeleteJobTest.php +│ ├── BulkActionPermissionTest.php +│ └── BulkEligibilityCheckTest.php +│ +└── Feature/ + ├── BulkDeletePoliciesTest.php + ├── BulkExportToBackupTest.php + ├── BulkProgressNotificationTest.php + └── BulkTypeToConfirmTest.php +``` + +**Structure Decision**: Single web application structure (Laravel + Filament). New bulk operations extend existing Resources with BulkAction definitions. New BulkOperationRun model tracks async job progress. No separate API layer needed (Livewire polling uses Filament infolists/resource pages). + +## Complexity Tracking + +> No constitution violations requiring justification. + +--- + +## Phase 0: Research & Technology Decisions + +See [research.md](./research.md) for detailed research findings. + +### Key Decisions Summary + +| Decision | Chosen | Rationale | +|----------|--------|-----------| +| Progress tracking | BulkOperationRun model + Livewire polling | Explicit state, survives page refresh, queryable outcomes | +| Job chunking | collect()->chunk(10) | Simple, memory-efficient, easy to test | +| Type-to-confirm | Filament form + validation rule | Built-in UI, reusable pattern | +| Tenant isolation | Explicit tenantId param | Fail-safe, auditable, no reliance on global scopes | +| Policy deletion | ignored_at flag | Prevents re-sync, restorable, doesn't touch Intune | +| Eligibility checks | Eloquent scopes | Reusable, testable, composable | + +--- + +## Phase 1: Data Model & Contracts + +See [data-model.md](./data-model.md) for detailed schemas and entity diagrams. + +### Core Entities + +**BulkOperationRun** (NEW): +- Tracks progress, outcomes, failures for bulk operations +- Fields: resource, action, status, total_items, processed_items, succeeded, failed, skipped +- JSONB: item_ids, failures +- Relationships: tenant, user, auditLog + +**Policy** (EXTEND): +- Add `ignored_at` timestamp (prevents re-sync) +- Add `markIgnored()` method and `notIgnored()` scope + +**PolicyVersion** (EXTEND): +- Add `pruneEligible()` scope (checks age, references, current status) + +**RestoreRun** (EXTEND): +- Add `deletable()` scope (filters by completed/failed status) + +--- + +## Phase 2: Implementation Tasks + +Detailed tasks will be generated via `/speckit.tasks` command. High-level phases: + +### Phase 2.1: Foundation (P1 - Policies) - 8-12 hours +- BulkOperationRun migration + model +- Policies: ignored_at column, bulk delete/export jobs +- Filament bulk actions + type-to-confirm +- BulkOperationService orchestration +- Tests (unit, feature) + +### Phase 2.2: Progress Tracking (P1) - 8-10 hours +- Livewire progress component +- Job progress updates (chunked) +- Circuit breaker (>50% fail abort) +- Audit logging integration +- Tests (progress, polling, audit) + +### Phase 2.3: Additional Resources (P2) - 6-8 hours +- PolicyVersion prune (eligibility scope) +- BackupSet bulk delete +- RestoreRun bulk delete +- Resource extensions +- Tests for each resource + +### Phase 2.4: Polish & Deployment - 4-6 hours +- Manual QA (type-to-confirm, progress UI) +- Load testing (500 items) +- Documentation updates +- Staging → Production deployment + +--- + +## Risk Mitigation + +| Risk | Mitigation | +|------|------------| +| Queue timeouts | Chunk processing (10-20 items), timeout config (300s), circuit breaker | +| Progress polling overhead | Limit interval (5s), index queries, cache recent runs | +| Accidental deletes | Type-to-confirm ≥20 items, `ignored_at` flag (restorable), audit trail | +| Job crashes | Fail-soft, BulkOperationRun status tracking, Laravel retry | +| Eligibility misses | Conservative JSONB queries, manual review before hard delete | +| Sync re-adds policies | `ignored_at` filter in SyncPoliciesJob | + +--- + +## Success Criteria + +- ✅ Bulk delete 100 policies in <2 minutes +- ✅ Type-to-confirm prevents accidents (≥20 items) +- ✅ Progress updates every 5-10s +- ✅ Audit log captures per-item outcomes +- ✅ 95%+ operation success rate +- ✅ All P1/P2 tests pass + +--- + +## Next Steps + +1. ✅ Generate plan.md (this file) +2. → Generate research.md (detailed technology findings) +3. → Generate data-model.md (schemas + diagrams) +4. → Generate quickstart.md (developer onboarding) +5. → Run `/speckit.tasks` to create task breakdown +6. → Begin Phase 2.1 implementation + +--- + +**Status**: Plan Complete - Ready for Research +**Created**: 2025-12-22 +**Last Updated**: 2025-12-22 diff --git a/specs/005-bulk-operations/quickstart.md b/specs/005-bulk-operations/quickstart.md new file mode 100644 index 0000000..000c74f --- /dev/null +++ b/specs/005-bulk-operations/quickstart.md @@ -0,0 +1,425 @@ +# Quickstart: Feature 005 - Bulk Operations + +**Feature**: Bulk Operations for Resource Management +**Date**: 2025-12-22 + +--- + +## Overview + +This quickstart guide helps developers get up and running with Feature 005 (Bulk Operations) for local development, testing, and debugging. + +--- + +## Prerequisites + +- Laravel Sail installed and running +- Composer dependencies installed +- NPM dependencies installed +- Database migrated +- At least one Tenant configured +- Sample Policies, PolicyVersions, BackupSets, RestoreRuns seeded + +--- + +## Local Development Setup + +### 1. Start Sail + +```bash +cd /path/to/TenantAtlas +./vendor/bin/sail up -d +``` + +### 2. Run Migrations + +```bash +./vendor/bin/sail artisan migrate +``` + +This creates: +- `bulk_operation_runs` table +- `ignored_at` column on `policies` table + +### 3. Seed Test Data (Optional) + +```bash +./vendor/bin/sail artisan db:seed --class=BulkOperationsTestSeeder +``` + +Creates: +- 100 test policies +- 50 policy versions (some old, some referenced) +- 10 backup sets +- 20 restore runs (mixed statuses) + +### 4. Start Queue Worker + +Bulk operations require queue processing: + +```bash +./vendor/bin/sail artisan queue:work --tries=3 --timeout=300 +``` + +Or run in background with Supervisor (production): + +```bash +./vendor/bin/sail artisan queue:restart +``` + +### 5. Access Filament Panel + +```bash +open http://localhost/admin +``` + +Navigate to: +- **Policies** → Select multiple → Bulk Actions dropdown +- **Policy Versions** → Bulk Prune +- **Backup Sets** → Bulk Delete +- **Restore Runs** → Bulk Delete + +--- + +## Running Tests + +### Unit Tests + +Test individual components (jobs, scopes, helpers): + +```bash +./vendor/bin/sail artisan test tests/Unit/BulkPolicyDeleteJobTest.php +./vendor/bin/sail artisan test tests/Unit/BulkActionPermissionTest.php +./vendor/bin/sail artisan test tests/Unit/BulkEligibilityCheckTest.php +``` + +### Feature Tests + +Test E2E flows (UI → job → database): + +```bash +./vendor/bin/sail artisan test tests/Feature/BulkDeletePoliciesTest.php +./vendor/bin/sail artisan test tests/Feature/BulkExportToBackupTest.php +./vendor/bin/sail artisan test tests/Feature/BulkProgressNotificationTest.php +./vendor/bin/sail artisan test tests/Feature/BulkTypeToConfirmTest.php +``` + +### All Tests + +```bash +./vendor/bin/sail artisan test --filter=Bulk +``` + +### Browser Tests (Pest v4) + +Test UI interactions: + +```bash +./vendor/bin/sail artisan test tests/Browser/BulkOperationsTest.php +``` + +--- + +## Manual Testing Workflow + +### Scenario 1: Bulk Delete Policies (< 20 items) + +1. **Navigate**: Admin → Policies +2. **Select**: Check 10 policies +3. **Action**: Click "Delete" in bulk actions dropdown +4. **Confirm**: Modal appears: "Delete 10 policies?" +5. **Submit**: Click "Confirm" +6. **Verify**: + - Success notification: "Deleted 10 policies" + - Policies have `ignored_at` timestamp set + - Policies still exist in Intune (no Graph DELETE call) + - Audit log entry created + +### Scenario 2: Bulk Delete Policies (≥ 20 items, queued) + +1. **Navigate**: Admin → Policies +2. **Select**: Check 25 policies +3. **Action**: Click "Delete" +4. **Confirm**: Modal requires typing "DELETE" +5. **Type**: Enter "DELETE" (case-sensitive) +6. **Submit**: Click "Confirm" +7. **Verify**: + - Job dispatched to queue + - Progress notification: "Deleting policies... 0/25" + - Notification updates every 5s: "Deleting... 10/25", "20/25" + - Final notification: "Deleted 25 policies" + - `BulkOperationRun` record created with status `completed` + - Audit log entry + +### Scenario 3: Bulk Export to Backup + +1. **Navigate**: Admin → Policies +2. **Select**: Check 30 policies +3. **Action**: Click "Export to Backup" +4. **Form**: + - Backup Set Name: "Production Snapshot" + - Include Assignments: ☑ (if Feature 004 implemented) +5. **Submit**: Click "Confirm" +6. **Verify**: + - Job dispatched + - Progress: "Backing up... 10/30" + - Final: "Backup Set 'Production Snapshot' created (30 items)" + - New `BackupSet` record + - 30 `BackupItem` records + - Audit log entry + +### Scenario 4: Bulk Prune Policy Versions + +1. **Navigate**: Admin → Policy Versions +2. **Filter**: Show only non-current versions older than 90 days +3. **Select**: Check 15 versions +4. **Action**: Click "Delete" +5. **Confirm**: Type "DELETE" +6. **Submit**: Click "Confirm" +7. **Verify**: + - Eligibility check runs + - Eligible versions deleted (hard delete) + - Ineligible versions skipped + - Notification: "Deleted 12 policy versions (3 skipped)" + - Failures array shows skip reasons: + - "Referenced in Backup Set #5" + - "Current version" + - "Too recent (< 90 days)" + +### Scenario 5: Circuit Breaker (abort on >50% fail) + +1. **Setup**: Mock Graph API to fail for 60% of items +2. **Navigate**: Admin → Policies +3. **Select**: Check 100 policies +4. **Action**: Bulk Delete +5. **Verify**: + - Job processes ~50 items + - Detects >50% failure rate + - Aborts remaining items + - Notification: "Bulk operation aborted: 55/100 failures exceeded threshold" + - `BulkOperationRun.status` = `aborted` + +--- + +## Debugging + +### View Queue Jobs + +```bash +# List failed jobs +./vendor/bin/sail artisan queue:failed + +# Retry failed job +./vendor/bin/sail artisan queue:retry + +# Flush failed jobs +./vendor/bin/sail artisan queue:flush +``` + +### Inspect BulkOperationRun Records + +```bash +./vendor/bin/sail tinker +``` + +```php +// Get recent runs +$runs = \App\Models\BulkOperationRun::recent()->limit(10)->get(); + +// View failures +$run = \App\Models\BulkOperationRun::find(1); +dd($run->failures); + +// Check progress +echo "{$run->processed_items}/{$run->total_items} ({$run->progressPercentage()}%)"; +``` + +### View Audit Logs + +```bash +# Via Filament UI +# Navigate to: Admin → Audit Logs → Filter by event: "policies.bulk_deleted" + +# Via Tinker +$logs = \App\Models\AuditLog::where('event', 'policies.bulk_deleted')->get(); +``` + +### Database Queries + +```bash +./vendor/bin/sail artisan tinker +``` + +```php +// Policies marked as ignored +$ignored = \App\Models\Policy::ignored()->count(); + +// Policy versions eligible for pruning +$eligible = \App\Models\PolicyVersion::pruneEligible(90)->count(); + +// Deletable restore runs +$deletable = \App\Models\RestoreRun::deletable()->count(); +``` + +### Test Queue Job Manually + +```bash +./vendor/bin/sail artisan tinker +``` + +```php +use App\Jobs\BulkPolicyDeleteJob; +use App\Models\BulkOperationRun; + +$policyIds = [1, 2, 3]; +$tenantId = 1; +$userId = 1; + +$run = BulkOperationRun::create([ + 'tenant_id' => $tenantId, + 'user_id' => $userId, + 'resource' => 'policies', + 'action' => 'delete', + 'status' => 'pending', + 'total_items' => count($policyIds), + 'item_ids' => $policyIds, +]); + +// Dispatch synchronously for debugging +BulkPolicyDeleteJob::dispatchSync($policyIds, $tenantId, $userId, $run->id); + +// Check result +$run->refresh(); +echo $run->summaryText(); +``` + +--- + +## Common Issues & Solutions + +### Issue 1: Type-to-confirm not working + +**Symptom**: Confirm button remains enabled without typing "DELETE" + +**Solution**: Check Filament form validation rule: +```php +->rule('in:DELETE') // Case-sensitive +``` + +### Issue 2: Progress notifications don't update + +**Symptom**: Progress stuck at "0/100" + +**Solution**: +- Ensure queue worker is running: `./vendor/bin/sail artisan queue:work` +- Check Livewire polling: `wire:poll.5s="refresh"` +- Verify BulkOperationRun is updated in job + +### Issue 3: Policies reappear after deletion + +**Symptom**: Deleted policies show up again after sync + +**Solution**: +- Check `ignored_at` is set: `Policy::find($id)->ignored_at` +- Verify SyncPoliciesJob filters: `->whereNull('ignored_at')` + +### Issue 4: Circuit breaker not aborting + +**Symptom**: Job continues despite >50% failures + +**Solution**: Check circuit breaker logic in job: +```php +if ($results['failed'] > count($this->policyIds) * 0.5) { + $run->update(['status' => 'aborted']); + throw new \Exception('Bulk operation aborted: >50% failure rate'); +} +``` + +### Issue 5: Policy versions deleted despite references + +**Symptom**: Referenced versions are deleted + +**Solution**: Verify eligibility scope includes: +```php +->whereDoesntHave('backupItems') +->whereNotIn('id', function ($subquery) { + $subquery->select(DB::raw("CAST(metadata->>'policy_version_id' AS INTEGER)")) + ->from('restore_runs') + ->whereNotNull(DB::raw("metadata->>'policy_version_id'")); +}); +``` + +--- + +## Performance Benchmarks + +Expected performance (local Sail environment): + +| Operation | Item Count | Duration | Notes | +|-----------|------------|----------|-------| +| Bulk Delete (sync) | 10 | <1s | Immediate feedback | +| Bulk Delete (queued) | 100 | <2min | Chunked, progress updates | +| Bulk Export | 50 | <3min | Includes Graph API calls | +| Bulk Prune | 30 | <30s | Eligibility checks | +| Progress Update | - | 5s | Polling interval | + +--- + +## Code Formatting + +Before committing: + +```bash +./vendor/bin/sail composer pint +``` + +Formats all PHP files per PSR-12. + +--- + +## Next Steps + +1. ✅ Complete Phase 2.1 (Foundation) tasks +2. ✅ Run all tests: `./vendor/bin/sail artisan test --filter=Bulk` +3. ✅ Manual QA: Follow scenarios above +4. ✅ Code review: Check PSR-12, permissions, audit logs +5. ✅ Load testing: Bulk delete 500 items +6. → Deploy to staging +7. → Manual QA on staging +8. → Deploy to production + +--- + +## Useful Commands + +```bash +# Watch queue jobs in real-time +./vendor/bin/sail artisan queue:work --verbose + +# Monitor bulk operations +./vendor/bin/sail artisan tinker +>>> BulkOperationRun::inProgress()->get() + +# Seed more test data +./vendor/bin/sail artisan db:seed --class=BulkOperationsTestSeeder + +# Clear cache +./vendor/bin/sail artisan optimize:clear + +# Restart queue workers (after code changes) +./vendor/bin/sail artisan queue:restart +``` + +--- + +## Resources + +- [Laravel Queue Documentation](https://laravel.com/docs/12.x/queues) +- [Filament Bulk Actions](https://filamentphp.com/docs/4.x/tables/actions#bulk-actions) +- [Livewire Polling](https://livewire.laravel.com/docs/polling) +- [Pest Testing](https://pestphp.com/docs) + +--- + +**Status**: Quickstart Complete +**Next Step**: Update agent context with new learnings diff --git a/specs/005-bulk-operations/research.md b/specs/005-bulk-operations/research.md new file mode 100644 index 0000000..78c0bc4 --- /dev/null +++ b/specs/005-bulk-operations/research.md @@ -0,0 +1,547 @@ +# Research: Feature 005 - Bulk Operations + +**Feature**: Bulk Operations for Resource Management +**Date**: 2025-12-22 +**Research Phase**: Technology Decisions & Best Practices + +--- + +## Research Questions & Findings + +### Q1: How to implement type-to-confirm in Filament bulk actions? + +**Research Goal**: Find a Laravel/Filament-idiomatic way to require explicit confirmation for destructive bulk operations (≥20 items). + +**Findings**: + +Filament BulkActions support conditional forms via `->form()` method: + +```php +Tables\Actions\DeleteBulkAction::make() + ->requiresConfirmation() + ->modalHeading(fn (Collection $records) => + $records->count() >= 20 + ? "⚠️ Delete {$records->count()} policies?" + : "Delete {$records->count()} policies?" + ) + ->form(fn (Collection $records) => + $records->count() >= 20 + ? [ + Forms\Components\TextInput::make('confirm_delete') + ->label('Type DELETE to confirm') + ->rule('in:DELETE') + ->required() + ->helperText('This action cannot be undone.') + ] + : [] + ) + ->action(fn (Collection $records, array $data) => { + // Validation ensures $data['confirm_delete'] === 'DELETE' + // Proceed with bulk delete + }); +``` + +**Key Insight**: Filament's form validation automatically prevents submission if `confirm_delete` doesn't match "DELETE" (case-sensitive). + +**Alternatives Considered**: +- Custom modal component (more code, less reusable) +- JavaScript validation (client-side only, less secure) +- Laravel form request (breaks Filament UX flow) + +**Decision**: Use Filament `->form()` with validation rule. + +--- + +### Q2: How to track progress for queued bulk jobs? + +**Research Goal**: Enable real-time progress tracking for async bulk operations (≥20 items) without blocking UI. + +**Findings**: + +Filament notifications are not reactive by default. Must implement custom progress tracking: + +1. **Create BulkOperationRun model** to persist state: + ```php + Schema::create('bulk_operation_runs', function (Blueprint $table) { + $table->id(); + $table->string('status'); // 'pending', 'running', 'completed', 'failed', 'aborted' + $table->integer('total_items'); + $table->integer('processed_items')->default(0); + $table->integer('succeeded')->default(0); + $table->integer('failed')->default(0); + $table->json('item_ids'); + $table->json('failures')->nullable(); + // ... tenant_id, user_id, resource, action + }); + ``` + +2. **Job updates model after each chunk**: + ```php + collect($this->policyIds)->chunk(10)->each(function ($chunk) use ($run) { + foreach ($chunk as $id) { + // Process item + } + $run->update([ + 'processed_items' => $run->processed_items + $chunk->count(), + // ... succeeded, failed counts + ]); + }); + ``` + +3. **UI polls for updates** via Livewire: + ```blade +
+ Processing... {{ $run->processed_items }}/{{ $run->total_items }} +
+ ``` + +**Alternatives Considered**: +- **Bus::batch()**: Laravel's batch system tracks job progress, but adds complexity: + - Requires job_batches table (already exists in Laravel) + - Each item becomes separate job (overhead for small batches) + - Good for parallelization, overkill for sequential processing + - Decision: **Not needed** - our jobs process items sequentially with chunking + +- **Filament Pulse**: Real-time application monitoring tool + - Too heavy for single-feature progress tracking + - Requires separate service + - Decision: **Rejected** - use custom BulkOperationRun model + +- **Pusher/WebSockets**: Real-time push notifications + - Infrastructure overhead (Pusher subscription or custom WS server) + - Not needed for 5-10s polling interval + - Decision: **Rejected** - Livewire polling sufficient + +**Decision**: BulkOperationRun model + Livewire polling (5s interval). + +--- + +### Q3: How to handle chunked processing in queue jobs? + +**Research Goal**: Process large batches (up to 500 items) without memory exhaustion or timeout. + +**Findings**: + +Laravel Collections provide `->chunk()` method for memory-efficient iteration: + +```php +collect($this->policyIds)->chunk(10)->each(function ($chunk) use (&$results, $run) { + foreach ($chunk as $id) { + try { + // Process item + $results['succeeded']++; + } catch (\Exception $e) { + $results['failed']++; + $results['failures'][] = ['id' => $id, 'reason' => $e->getMessage()]; + } + } + + // Update progress after each chunk (not per-item) + $run->update([ + 'processed_items' => $results['succeeded'] + $results['failed'], + 'succeeded' => $results['succeeded'], + 'failed' => $results['failed'], + 'failures' => $results['failures'], + ]); + + // Circuit breaker: abort if >50% failed + if ($results['failed'] > count($this->policyIds) * 0.5) { + $run->update(['status' => 'aborted']); + throw new \Exception('Bulk operation aborted: >50% failure rate'); + } +}); +``` + +**Key Insights**: +- Chunk size: 10-20 items (balance between DB updates and progress granularity) +- Update BulkOperationRun **after each chunk**, not per-item (reduces DB load) +- Circuit breaker: abort if >50% failures detected mid-process +- Fail-soft: continue processing remaining items on individual failures + +**Alternatives Considered**: +- **Cursor-based chunking**: `Model::chunk(100, function)` + - Good for processing entire tables + - Not needed - we have explicit ID list + +- **Bus::batch()**: Parallel job processing + - Good for independent tasks (e.g., sending emails) + - Our tasks are sequential (delete one, then next) + - Adds complexity without benefit + +- **Database transactions per chunk**: + - Risk: partial failure leaves incomplete state + - Decision: **No transactions** - each item is atomic, fail-soft is intentional + +**Decision**: `collect()->chunk(10)` with after-chunk progress updates. + +--- + +### Q4: How to enforce tenant isolation in bulk jobs? + +**Research Goal**: Ensure bulk operations cannot cross tenant boundaries (critical security requirement). + +**Findings**: + +Laravel Queue jobs serialize model instances poorly (especially Collections). Best practice: + +```php +class BulkPolicyDeleteJob implements ShouldQueue +{ + public function __construct( + public array $policyIds, // array, NOT Collection + public int $tenantId, // explicit tenant ID + public int $actorId, // user ID for audit + public int $bulkOperationRunId // FK to tracking model + ) {} + + public function handle(PolicyRepository $policies): void + { + // Verify all policies belong to tenant (defensive check) + $count = Policy::whereIn('id', $this->policyIds) + ->where('tenant_id', $this->tenantId) + ->count(); + + if ($count !== count($this->policyIds)) { + throw new \Exception('Tenant isolation violation detected'); + } + + // Proceed with bulk operation... + } +} +``` + +**Key Insights**: +- Serialize IDs as `array`, not `Collection` (Collections don't serialize well) +- Pass explicit `tenantId` parameter (don't rely on global scopes) +- Defensive check in job: verify all IDs belong to tenant before processing +- Audit log records `tenantId` and `actorId` for compliance + +**Alternatives Considered**: +- **Global tenant scope**: Rely on Laravel's global scope filtering + - Risk: scope could be disabled/bypassed in job context + - Less explicit, harder to debug + - Decision: **Rejected** - explicit is safer + +- **Pass User model**: `public User $user` + - Serializes entire user object (inefficient) + - User could be deleted before job runs + - Decision: **Rejected** - use `actorId` integer + +**Decision**: Explicit `tenantId` + defensive validation in job. + +--- + +### Q5: How to prevent sync from re-adding "deleted" policies? + +**Research Goal**: User bulk-deletes 50 policies locally, but doesn't want to delete them in Intune. How to prevent SyncPoliciesJob from re-importing them? + +**Findings**: + +Add `ignored_at` timestamp column to policies table: + +```php +// Migration +Schema::table('policies', function (Blueprint $table) { + $table->timestamp('ignored_at')->nullable()->after('deleted_at'); + $table->index('ignored_at'); // query optimization +}); + +// Policy model +public function scopeNotIgnored($query) +{ + return $query->whereNull('ignored_at'); +} + +public function markIgnored(): void +{ + $this->update(['ignored_at' => now()]); +} +``` + +**Modify SyncPoliciesJob**: + +```php +// Before: fetched all policies from Graph, upserted to DB +// After: skip policies where ignored_at IS NOT NULL + +public function handle(PolicySyncService $service): void +{ + $graphPolicies = $service->fetchFromGraph($this->types); + + foreach ($graphPolicies as $graphPolicy) { + $existing = Policy::where('graph_id', $graphPolicy['id']) + ->where('tenant_id', $this->tenantId) + ->first(); + + // Skip if locally ignored + if ($existing && $existing->ignored_at !== null) { + continue; + } + + // Upsert policy... + } +} +``` + +**Key Insight**: `ignored_at` decouples local tracking from Intune state. User can: +- Keep policy in Intune (not deleted remotely) +- Hide policy in TenantPilot (ignored_at set) +- Restore policy later (clear ignored_at) + +**Alternatives Considered**: +- **Soft delete only** (`deleted_at`): + - Problem: Sync doesn't know if user deleted locally or Intune deleted remotely + - Would need separate "deletion source" column + - Decision: **Rejected** - `ignored_at` is clearer intent + +- **Separate "sync_ignore" column**: + - Same outcome as `ignored_at`, but less semantic + - Decision: **Accepted as alias** - `ignored_at` is more descriptive + +**Decision**: Add `ignored_at` timestamp, filter in SyncPoliciesJob. + +--- + +### Q6: How to determine eligibility for Policy Version pruning? + +**Research Goal**: Implement safe "bulk delete old policy versions" that won't break backups/restores. + +**Findings**: + +Eligibility criteria (all must be true): +1. `is_current = false` (not the latest version) +2. `created_at < NOW() - 90 days` (configurable retention period) +3. NOT referenced in `backup_items.policy_version_id` (foreign key check) +4. NOT referenced in `restore_runs.metadata->policy_version_id` (JSONB check) + +Implementation via Eloquent scope: + +```php +// app/Models/PolicyVersion.php +public function scopePruneEligible($query, int $retentionDays = 90) +{ + return $query + ->where('is_current', false) + ->where('created_at', '<', now()->subDays($retentionDays)) + ->whereDoesntHave('backupItems') // FK relationship + ->whereNotIn('id', function ($subquery) { + $subquery->select(DB::raw("CAST(metadata->>'policy_version_id' AS INTEGER)")) + ->from('restore_runs') + ->whereNotNull(DB::raw("metadata->>'policy_version_id'")); + }); +} +``` + +**Bulk prune job**: + +```php +public function handle(): void +{ + foreach ($this->versionIds as $id) { + $version = PolicyVersion::find($id); + + if (!$version) { + $this->failures[] = ['id' => $id, 'reason' => 'Not found']; + continue; + } + + // Check eligibility + $eligible = PolicyVersion::pruneEligible() + ->where('id', $id) + ->exists(); + + if (!$eligible) { + $this->skipped++; + $this->failures[] = ['id' => $id, 'reason' => 'Referenced or too recent']; + continue; + } + + $version->delete(); // hard delete + $this->succeeded++; + } +} +``` + +**Key Insight**: Conservative eligibility check prevents accidental data loss. User sees which versions were skipped and why. + +**Alternatives Considered**: +- **Soft delete first, hard delete later**: Adds complexity, no clear benefit +- **Skip JSONB check**: Risk of breaking restore runs that reference version +- **Admin override**: Allow force-delete even if referenced + - Too dangerous, conflicts with immutability principle + - Decision: **Rejected** + +**Decision**: Eloquent scope `pruneEligible()` with strict checks. + +--- + +### Q7: How to display progress notifications in Filament? + +**Research Goal**: Show real-time progress for bulk operations without blocking UI. + +**Findings**: + +Filament notifications are sent once and don't auto-update. For progress tracking: + +**Option 1: Custom Livewire Component** + +```blade +{{-- resources/views/livewire/bulk-operation-progress.blade.php --}} +
+ @if($run && !$run->isComplete()) +
+

{{ $run->action }} in progress...

+
+
+
+

{{ $run->processed_items }}/{{ $run->total_items }} items processed

+
+ @elseif($run && $run->isComplete()) +
+

✅ {{ $run->summaryText() }}

+ @if($run->failed > 0) + View details + @endif +
+ @endif +
+``` + +```php +// app/Livewire/BulkOperationProgress.php +class BulkOperationProgress extends Component +{ + public int $bulkOperationRunId; + public ?BulkOperationRun $run = null; + + public function mount(int $bulkOperationRunId): void + { + $this->bulkOperationRunId = $bulkOperationRunId; + $this->refresh(); + } + + public function refresh(): void + { + $this->run = BulkOperationRun::find($this->bulkOperationRunId); + + // Stop polling if complete + if ($this->run && $this->run->isComplete()) { + $this->dispatch('bulkOperationComplete', runId: $this->run->id); + } + } + + public function render(): View + { + return view('livewire.bulk-operation-progress'); + } +} +``` + +**Option 2: Filament Infolist Widget** (simpler, more integrated) + +```php +// Display in BulkOperationRun resource ViewRecord page +public static function form(Form $form): Form +{ + return $form + ->schema([ + Infolists\Components\Section::make('Progress') + ->schema([ + Infolists\Components\TextEntry::make('summaryText') + ->label('Status'), + Infolists\Components\ViewEntry::make('progress') + ->view('filament.components.progress-bar') + ->state(fn ($record) => [ + 'percentage' => $record->progressPercentage(), + 'processed' => $record->processed_items, + 'total' => $record->total_items, + ]), + ]) + ->poll('5s') // Filament's built-in polling + ->hidden(fn ($record) => $record->isComplete()), + ]); +} +``` + +**Decision**: Use **Option 1** (custom Livewire component) for flexibility. Embed in: +- Filament notification body (custom view) +- Resource page sidebar +- Dashboard widget (if user wants to monitor all bulk operations) + +**Alternatives Considered**: +- **Pusher/WebSockets**: Too complex for 5s polling +- **JavaScript polling**: Less Laravel-way, harder to test +- **Filament Pulse**: Overkill for single feature + +--- + +## Technology Stack Summary + +| Component | Technology | Justification | +|-----------|------------|---------------| +| Admin Panel | Filament v4 | Built-in bulk actions, forms, notifications | +| Reactive UI | Livewire v3 | Polling, state management, no JS framework needed | +| Queue System | Laravel Queue | Async job processing, retry, failure handling | +| Progress Tracking | BulkOperationRun model + Livewire polling | Persistent state, survives refresh, queryable | +| Type-to-Confirm | Filament form validation | Built-in UI, secure, reusable | +| Tenant Isolation | Explicit tenantId param | Fail-safe, auditable, no implicit scopes | +| Job Chunking | Collection::chunk(10) | Memory-efficient, simple, testable | +| Eligibility Checks | Eloquent scopes | Reusable, composable, database-level filtering | +| Database | PostgreSQL + JSONB | Native JSON support for item_ids, failures | + +--- + +## Best Practices Applied + +### Laravel Conventions +- ✅ Queue jobs implement `ShouldQueue` interface +- ✅ Use Eloquent relationships, not raw queries +- ✅ Form validation via Filament rules +- ✅ PSR-12 code formatting (Laravel Pint) + +### Safety & Security +- ✅ Tenant isolation enforced at job level +- ✅ Type-to-confirm for ≥20 destructive items +- ✅ Fail-soft: continue on individual failures +- ✅ Circuit breaker: abort if >50% fail +- ✅ Audit logging for compliance + +### Performance +- ✅ Chunked processing (10-20 items) +- ✅ Indexed queries (tenant_id, ignored_at) +- ✅ Polling interval: 5s (not 1s spam) +- ✅ JSONB for flexible metadata storage + +### Testing +- ✅ Unit tests for jobs, scopes, eligibility +- ✅ Feature tests for E2E flows +- ✅ Pest assertions for progress tracking +- ✅ Manual QA checklist for UI flows + +--- + +## Rejected Alternatives + +| Alternative | Why Rejected | +|-------------|--------------| +| Bus::batch() | Adds complexity, not needed for sequential processing | +| Filament Pulse | Overkill for single-feature progress tracking | +| Pusher/WebSockets | Infrastructure overhead, 5s polling sufficient | +| Global tenant scopes | Less explicit, harder to debug, security risk | +| Custom modal component | More code, less reusable than Filament form | +| Hard delete without checks | Too risky, violates immutability principle | + +--- + +## Open Questions for Implementation + +1. **Chunk size**: Start with 10, benchmark if needed +2. **Polling interval**: 5s default, make configurable? +3. **Retention period**: 90 days for versions, make configurable? +4. **Max bulk items**: Hard limit at 500? 1000? +5. **Retry failed items**: Future enhancement or MVP? + +--- + +**Status**: Research Complete +**Next Step**: Generate data-model.md