TenantAtlas/specs/005-bulk-operations/data-model.md

Data Model: Feature 005 - Bulk Operations

Feature: Bulk Operations for Resource Management
Date: 2025-12-22

---

Overview

This document describes the data model for bulk operations, including new entities, schema changes, relationships, and query patterns.

---

Entity Diagram

┌─────────────────────────────────────────────────────────────┐
│                      BulkOperationRun                       │
├─────────────────────────────────────────────────────────────┤
│ id: bigint PK                                               │
│ tenant_id: bigint FK → Tenant                               │
│ user_id: bigint FK → User                                   │
│ resource: string (policies, policy_versions, etc.)          │
│ action: string (delete, export, prune, etc.)                │
│ status: enum (pending, running, completed, failed, aborted) │
│ total_items: int                                            │
│ processed_items: int                                        │
│ succeeded: int                                              │
│ failed: int                                                 │
│ skipped: int                                                │
│ item_ids: jsonb (array of IDs)                             │
│ failures: jsonb (array of {id, reason})                    │
│ audit_log_id: bigint FK → AuditLog (nullable)              │
│ created_at: timestamp                                       │
│ updated_at: timestamp                                       │
└─────────────────────────────────────────────────────────────┘
         │             │             │
         │             │             └─────────────┐
         │             │                           │
         ▼             ▼                           ▼
┌──────────────┐ ┌──────────────┐         ┌──────────────┐
│    Tenant    │ │     User     │         │  AuditLog    │
└──────────────┘ └──────────────┘         └──────────────┘


┌─────────────────────────────────────────┐
│            Policy (Extended)            │
├─────────────────────────────────────────┤
│ id: bigint PK                           │
│ tenant_id: bigint FK                    │
│ graph_id: string                        │
│ name: string                            │
│ ... (existing columns)                  │
│ deleted_at: timestamp (nullable)        │
│ ignored_at: timestamp (nullable) ← NEW  │
└─────────────────────────────────────────┘


┌─────────────────────────────────────────┐
│       PolicyVersion (Extended)          │
├─────────────────────────────────────────┤
│ id: bigint PK                           │
│ policy_id: bigint FK → Policy           │
│ is_current: boolean                     │
│ created_at: timestamp                   │
│ ... (existing columns)                  │
├─────────────────────────────────────────┤
│ Scope: pruneEligible() ← NEW            │
│   WHERE is_current = false              │
│   AND created_at < NOW() - 90 days      │
│   AND NOT IN (backup_items)             │
│   AND NOT IN (restore_runs.metadata)    │
└─────────────────────────────────────────┘


┌─────────────────────────────────────────┐
│        RestoreRun (Extended)            │
├─────────────────────────────────────────┤
│ id: bigint PK                           │
│ status: enum (pending, running, ...)    │
│ ... (existing columns)                  │
├─────────────────────────────────────────┤
│ Scope: deletable() ← NEW                │
│   WHERE status IN (completed, failed)   │
└─────────────────────────────────────────┘

---

Database Schema

New Table: bulk_operation_runs

CREATE TABLE bulk_operation_runs (
    id BIGSERIAL PRIMARY KEY,
    tenant_id BIGINT NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,
    user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    resource VARCHAR(50) NOT NULL, -- 'policies', 'policy_versions', 'backup_sets', 'restore_runs'
    action VARCHAR(50) NOT NULL,   -- 'delete', 'export', 'prune', 'archive'
    status VARCHAR(20) NOT NULL,   -- 'pending', 'running', 'completed', 'failed', 'aborted'
    total_items INT NOT NULL,
    processed_items INT NOT NULL DEFAULT 0,
    succeeded INT NOT NULL DEFAULT 0,
    failed INT NOT NULL DEFAULT 0,
    skipped INT NOT NULL DEFAULT 0,
    item_ids JSONB NOT NULL,       -- [1, 2, 3, ...]
    failures JSONB,                -- [{"id": 1, "reason": "Graph error"}, ...]
    audit_log_id BIGINT REFERENCES audit_logs(id) ON DELETE SET NULL,
    created_at TIMESTAMP NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_bulk_operation_runs_tenant_resource_status 
    ON bulk_operation_runs(tenant_id, resource, status);

CREATE INDEX idx_bulk_operation_runs_user_created 
    ON bulk_operation_runs(user_id, created_at DESC);

CREATE INDEX idx_bulk_operation_runs_status 
    ON bulk_operation_runs(status) 
    WHERE status IN ('pending', 'running');

Indexes Rationale:

• Composite (tenant_id, resource, status): Filter by tenant + resource type + status (UI queries)
• (user_id, created_at): User's recent bulk operations
• Partial index on status: Only index running/pending (99% of queries check these)

---

Schema Change: policies table

ALTER TABLE policies 
ADD COLUMN ignored_at TIMESTAMP NULL;

CREATE INDEX idx_policies_ignored_at 
    ON policies(ignored_at) 
    WHERE ignored_at IS NOT NULL;

Purpose: Prevent SyncPoliciesJob from re-importing locally deleted policies.

Query Pattern:

-- Sync job filters
SELECT * FROM policies WHERE ignored_at IS NULL;

-- Bulk delete sets
UPDATE policies SET ignored_at = NOW() WHERE id IN (...);

---

Schema Change: policy_versions table

No schema changes needed. Eligibility scope uses existing columns:

• is_current (boolean)
• created_at (timestamp)
• Foreign keys checked via relationships

---

Eloquent Models

New Model: BulkOperationRun

<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Relations\BelongsTo;

class BulkOperationRun extends Model
{
    protected $guarded = [];

    protected $casts = [
        'item_ids' => 'array',
        'failures' => 'array',
        'created_at' => 'datetime',
        'updated_at' => 'datetime',
    ];

    // Relationships
    
    public function tenant(): BelongsTo
    {
        return $this->belongsTo(Tenant::class);
    }

    public function user(): BelongsTo
    {
        return $this->belongsTo(User::class);
    }

    public function auditLog(): BelongsTo
    {
        return $this->belongsTo(AuditLog::class);
    }

    // Status Helpers
    
    public function isRunning(): bool
    {
        return $this->status === 'running';
    }

    public function isComplete(): bool
    {
        return in_array($this->status, ['completed', 'failed', 'aborted']);
    }

    public function isPending(): bool
    {
        return $this->status === 'pending';
    }

    // Progress Helpers
    
    public function progressPercentage(): int
    {
        if ($this->total_items === 0) {
            return 0;
        }
        return (int) round(($this->processed_items / $this->total_items) * 100);
    }

    public function summaryText(): string
    {
        return match ($this->status) {
            'pending' => "Pending: {$this->total_items} items",
            'running' => "Processing... {$this->processed_items}/{$this->total_items}",
            'completed' => "Completed: {$this->succeeded} succeeded, {$this->failed} failed, {$this->skipped} skipped",
            'failed' => "Failed: {$this->failed}/{$this->total_items} errors",
            'aborted' => "Aborted: Too many failures ({$this->failed}/{$this->total_items})",
            default => "Unknown status",
        };
    }

    public function hasFailures(): bool
    {
        return $this->failed > 0;
    }

    // Scopes
    
    public function scopeForResource($query, string $resource)
    {
        return $query->where('resource', $resource);
    }

    public function scopeForUser($query, int $userId)
    {
        return $query->where('user_id', $userId);
    }

    public function scopeRecent($query)
    {
        return $query->orderBy('created_at', 'desc');
    }

    public function scopeInProgress($query)
    {
        return $query->whereIn('status', ['pending', 'running']);
    }
}

---

Extended Model: Policy

<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\SoftDeletes;

class Policy extends Model
{
    use SoftDeletes;

    // ... existing code ...

    // NEW: Scopes for bulk operations
    
    public function scopeNotIgnored($query)
    {
        return $query->whereNull('ignored_at');
    }

    public function scopeIgnored($query)
    {
        return $query->whereNotNull('ignored_at');
    }

    // NEW: Methods for bulk operations
    
    public function markIgnored(): void
    {
        $this->update(['ignored_at' => now()]);
    }

    public function unmarkIgnored(): void
    {
        $this->update(['ignored_at' => null]);
    }

    public function isIgnored(): bool
    {
        return $this->ignored_at !== null;
    }
}

---

Extended Model: PolicyVersion

<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Model;
use Illuminate\Support\Facades\DB;

class PolicyVersion extends Model
{
    // ... existing code ...

    // Relationships (if not already defined)
    
    public function backupItems()
    {
        return $this->hasMany(BackupItem::class, 'policy_version_id');
    }

    // NEW: Scope for eligibility check
    
    public function scopePruneEligible($query, int $retentionDays = 90)
    {
        return $query
            ->where('is_current', false)
            ->where('created_at', '<', now()->subDays($retentionDays))
            ->whereDoesntHave('backupItems')
            ->whereNotIn('id', function ($subquery) {
                $subquery->select(DB::raw("CAST(metadata->>'policy_version_id' AS INTEGER)"))
                    ->from('restore_runs')
                    ->whereNotNull(DB::raw("metadata->>'policy_version_id'"));
            });
    }

    // NEW: Check if version is eligible for pruning
    
    public function isPruneEligible(int $retentionDays = 90): bool
    {
        if ($this->is_current) {
            return false;
        }

        if ($this->created_at->diffInDays(now()) < $retentionDays) {
            return false;
        }

        if ($this->backupItems()->exists()) {
            return false;
        }

        $referencedInRestoreRuns = DB::table('restore_runs')
            ->whereNotNull(DB::raw("metadata->>'policy_version_id'"))
            ->where(DB::raw("CAST(metadata->>'policy_version_id' AS INTEGER)"), $this->id)
            ->exists();

        return !$referencedInRestoreRuns;
    }

    // NEW: Get reason why version is not eligible
    
    public function getIneligibilityReason(int $retentionDays = 90): ?string
    {
        if ($this->is_current) {
            return 'Current version';
        }

        if ($this->created_at->diffInDays(now()) < $retentionDays) {
            return "Too recent (< {$retentionDays} days)";
        }

        if ($this->backupItems()->exists()) {
            return 'Referenced in backup items';
        }

        $referencedInRestoreRuns = DB::table('restore_runs')
            ->whereNotNull(DB::raw("metadata->>'policy_version_id'"))
            ->where(DB::raw("CAST(metadata->>'policy_version_id' AS INTEGER)"), $this->id)
            ->exists();

        if ($referencedInRestoreRuns) {
            return 'Referenced in restore runs';
        }

        return null; // Eligible
    }
}

---

Extended Model: RestoreRun

<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Model;

class RestoreRun extends Model
{
    // ... existing code ...

    // NEW: Scope for deletable restore runs
    
    public function scopeDeletable($query)
    {
        return $query->whereIn('status', ['completed', 'failed', 'aborted']);
    }

    public function scopeNotDeletable($query)
    {
        return $query->whereNotIn('status', ['completed', 'failed', 'aborted']);
    }

    // NEW: Check if restore run can be deleted
    
    public function isDeletable(): bool
    {
        return in_array($this->status, ['completed', 'failed', 'aborted']);
    }

    public function getNotDeletableReason(): ?string
    {
        if ($this->isDeletable()) {
            return null;
        }

        return match ($this->status) {
            'pending' => 'Restore run is pending',
            'running' => 'Restore run is currently running',
            default => "Restore run has status: {$this->status}",
        };
    }
}

---

Query Patterns

1. Create Bulk Operation Run

$run = BulkOperationRun::create([
    'tenant_id' => $tenantId,
    'user_id' => $userId,
    'resource' => 'policies',
    'action' => 'delete',
    'status' => 'pending',
    'total_items' => count($policyIds),
    'item_ids' => $policyIds,
]);

// Dispatch job
BulkPolicyDeleteJob::dispatch($policyIds, $tenantId, $userId, $run->id);

$run->update(['status' => 'running']);

2. Update Progress (in job)

$run->update([
    'processed_items' => $run->processed_items + $chunkSize,
    'succeeded' => $run->succeeded + $successCount,
    'failed' => $run->failed + $failCount,
    'failures' => array_merge($run->failures ?? [], $newFailures),
]);

3. Query Recent Bulk Operations (UI)

// User's recent operations
$runs = BulkOperationRun::forUser(auth()->id())
    ->recent()
    ->limit(10)
    ->get();

// Tenant's in-progress operations
$inProgress = BulkOperationRun::where('tenant_id', $tenantId)
    ->inProgress()
    ->get();

4. Filter Policies (exclude ignored)

// Sync job
$policies = Policy::where('tenant_id', $tenantId)
    ->notIgnored()
    ->get();

// Bulk delete (mark as ignored)
Policy::whereIn('id', $policyIds)
    ->update(['ignored_at' => now()]);

5. Check Policy Version Eligibility

// Get all eligible versions for tenant
$eligibleVersions = PolicyVersion::whereHas('policy', function ($query) use ($tenantId) {
        $query->where('tenant_id', $tenantId);
    })
    ->pruneEligible(90)
    ->get();

// Check single version
$version = PolicyVersion::find($id);
if (!$version->isPruneEligible()) {
    $reason = $version->getIneligibilityReason();
    // Skip with reason: "Referenced in backup items"
}

6. Filter Deletable Restore Runs

// Get deletable runs
$deletableRuns = RestoreRun::where('tenant_id', $tenantId)
    ->deletable()
    ->get();

// Check individual run
$run = RestoreRun::find($id);
if (!$run->isDeletable()) {
    $reason = $run->getNotDeletableReason();
    // Skip: "Restore run is currently running"
}

---

JSONB Structure

bulk_operation_runs.item_ids

[1, 2, 3, 4, 5, ...]

Simple array of integer IDs.

bulk_operation_runs.failures

[
  {
    "id": 123,
    "reason": "Graph API error: 503 Service Unavailable"
  },
  {
    "id": 456,
    "reason": "Policy not found"
  },
  {
    "id": 789,
    "reason": "Permission denied"
  }
]

Array of objects with id (resource ID) and reason (error message).

restore_runs.metadata (existing)

{
  "policy_version_id": 42,
  "backup_set_id": 15,
  "items_count": 10,
  ...
}

When checking eligibility, query:

SELECT * FROM restore_runs 
WHERE metadata->>'policy_version_id' = '42';

---

Migration Files

Migration 1: Create bulk_operation_runs table

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function up(): void
    {
        Schema::create('bulk_operation_runs', function (Blueprint $table) {
            $table->id();
            $table->foreignId('tenant_id')->constrained()->cascadeOnDelete();
            $table->foreignId('user_id')->constrained()->cascadeOnDelete();
            $table->string('resource', 50);
            $table->string('action', 50);
            $table->string('status', 20);
            $table->integer('total_items');
            $table->integer('processed_items')->default(0);
            $table->integer('succeeded')->default(0);
            $table->integer('failed')->default(0);
            $table->integer('skipped')->default(0);
            $table->json('item_ids');
            $table->json('failures')->nullable();
            $table->foreignId('audit_log_id')->nullable()->constrained()->nullOnDelete();
            $table->timestamps();

            $table->index(['tenant_id', 'resource', 'status']);
            $table->index(['user_id', 'created_at']);
            $table->index('status')->where('status', 'in', ['pending', 'running']);
        });
    }

    public function down(): void
    {
        Schema::dropIfExists('bulk_operation_runs');
    }
};

Migration 2: Add ignored_at to policies

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function up(): void
    {
        Schema::table('policies', function (Blueprint $table) {
            $table->timestamp('ignored_at')->nullable()->after('deleted_at');
            $table->index('ignored_at');
        });
    }

    public function down(): void
    {
        Schema::table('policies', function (Blueprint $table) {
            $table->dropIndex(['ignored_at']);
            $table->dropColumn('ignored_at');
        });
    }
};

---

Data Retention & Cleanup

BulkOperationRun Retention

Recommended: Keep for 90 days, then archive or delete completed runs.

// Scheduled command
Artisan::command('bulk-operations:prune', function () {
    $deleted = BulkOperationRun::where('created_at', '<', now()->subDays(90))
        ->whereIn('status', ['completed', 'failed', 'aborted'])
        ->delete();

    $this->info("Pruned {$deleted} old bulk operation runs");
})->daily();

PolicyVersion Retention

Handled by bulk prune feature (user-initiated, not automatic).

---

Status: Data Model Complete
Next Step: Generate quickstart.md