# Tasks: Backend Architecture Pivot **Feature**: 005-backend-arch-pivot **Generated**: 2025-12-09 **Total Tasks**: 66 (T001-T066) **Spec**: [spec.md](./spec.md) | **Plan**: [plan.md](./plan.md) ## Phase 1: Setup (no story label) - [ ] T001 Confirm Dokploy-provided `REDIS_URL` and record connection string in `specs/005-backend-arch-pivot/notes.md` - [ ] T002 Add `REDIS_URL` to local `.env.example` and project `.env` (if used) (`.env.example`) - [ ] T003 Update `lib/env.mjs` to validate `REDIS_URL` (`lib/env.mjs`) - [ ] T004 [P] Add npm dependencies: `bullmq`, `ioredis`, `@azure/identity` and dev `tsx` (`package.json`) - [ ] T005 [P] Add npm script `worker:start` to `package.json` to run `tsx ./worker/index.ts` (`package.json`) - [X] T006 [P] Create `lib/queue/redis.ts` - Redis connection wrapper reading `process.env.REDIS_URL` (`lib/queue/redis.ts`) - [X] T007 [P] Create `lib/queue/syncQueue.ts` - Export BullMQ `Queue('intune-sync-queue')` (`lib/queue/syncQueue.ts`) - [X] T008 Test connectivity: add a dummy job from a Node REPL/script and verify connection to provided Redis (`scripts/test-queue-connection.js`) ## Phase 2: Worker Skeleton (no story label) - [ ] T009 Create `worker/index.ts` - minimal BullMQ `Worker` entry point (concurrency:1) (`worker/index.ts`) - [ ] T010 Create `worker/logging.ts` - structured JSON logger used by worker (`worker/logging.ts`) - [ ] T011 Create `worker/events.ts` - job lifecycle event handlers (completed/failed) (`worker/events.ts`) - [ ] T012 [P] Add `npm run worker:start` integration to `README.md` with run instructions (`README.md`) - [ ] T013 Create `worker/health.ts` - minimal health check handlers (used in docs) (`worker/health.ts`) - [ ] T014 Smoke test: start `npm run worker:start` and verify worker connects and logs idle state (no file) ## Phase 3: US1 — Manual Policy Sync via Queue [US1] - [ ] T015 [US1] Update `lib/actions/policySettings.ts` → implement `triggerPolicySync()` to call `syncQueue.add(...)` and return `jobId` (`lib/actions/policySettings.ts`) - [ ] T016 [US1] Create server action wrapper if needed `app/actions/triggerPolicySync.ts` (`app/actions/triggerPolicySync.ts`) - [ ] T017 [US1] Update `/app/search/SyncButton.tsx` to call server action and show queued toast with `jobId` (`components/search/SyncButton.tsx`) - [ ] T018 [US1] Add API route `/api/policy-sync/status` (optional) to report job status using BullMQ `Job` API (`app/api/policy-sync/status/route.ts`) - [ ] T019 [US1] Add simple job payload typing `types/syncJob.ts` (`types/syncJob.ts`) - [ ] T020 [US1] Add unit test for `triggerPolicySync()` mocking `syncQueue.add` (`tests/unit/triggerPolicySync.test.ts`) - [ ] T021 [US1] End-to-end test: UI → triggerPolicySync → job queued (integration test) (`tests/e2e/sync-button.test.ts`) - [ ] T022 [US1] OPTIONAL [P] Document MVP scope for job status endpoint (FR-022) in `specs/005-backend-arch-pivot/notes.md` (`specs/005-backend-arch-pivot/notes.md`) ## Phase 4: US2 — Microsoft Graph Data Fetching [US2] - [ ] T023 [US2] Create `worker/jobs/graphAuth.ts` - `getGraphAccessToken()` using `@azure/identity` (`worker/jobs/graphAuth.ts`) - [ ] T024 [US2] Create `worker/jobs/graphFetch.ts` - `fetchFromGraph(endpoint)` with pagination following `@odata.nextLink` (`worker/jobs/graphFetch.ts`) - [ ] T025 [US2] Implement `worker/utils/retry.ts` - exponential backoff retry helper (`worker/utils/retry.ts`) - [ ] T026 [US2] Create integration tests mocking Graph endpoints for paginated responses (`tests/integration/graphFetch.test.ts`) - [ ] T027 [US2] Implement rate limit handling and transient error classification in `graphFetch.ts` (`worker/jobs/graphFetch.ts`) - [ ] T028 [US2] Add logging for Graph fetch metrics (requests, pages, duration) (`worker/logging.ts`) - [ ] T029 [US2] Test: run `syncPolicies` job locally against mocked Graph responses (`tests/e2e/sync-with-mock-graph.test.ts`) ## Phase 5: US3 — Deep Flattening & Transformation [US3] - [ ] T030 [US3] Create `worker/jobs/policyParser.ts` - top-level router and `parsePolicySettings()` (`worker/jobs/policyParser.ts`) - [ ] T031 [US3] Implement Settings Catalog parser in `policyParser.ts` (`worker/jobs/policyParser.ts`) - [ ] T032 [US3] Implement OMA-URI parser in `policyParser.ts` (`worker/jobs/policyParser.ts`) - [ ] T033 [US3] Create `worker/utils/humanizer.ts` - `humanizeSettingId()` function (`worker/utils/humanizer.ts`) - [ ] T034 [US3] Create normalization function `worker/jobs/normalizer.ts` to produce `PolicyInsertData[]` (`worker/jobs/normalizer.ts`) - [ ] T035 [US3] Unit tests for parsers + humanizer with representative Graph samples (`tests/unit/policyParser.test.ts`) ## Phase 6: US3 — Database Persistence (shared, assign to US3) [US3] - [ ] T036 [US3] Create `worker/jobs/dbUpsert.ts` - batch upsert function using Drizzle (`worker/jobs/dbUpsert.ts`) - [ ] T037 [US3] Implement transactional upsert logic and `ON CONFLICT DO UPDATE` behavior (`worker/jobs/dbUpsert.ts`) - [ ] T038 [US3] Add performance tuning: batch size config and bulk insert strategy (`worker/jobs/dbUpsert.ts`) - [ ] T039 [US3] Add tests for upsert correctness (duplicates / conflict resolution) (`tests/integration/dbUpsert.test.ts`) - [ ] T040 [US3] Add `lastSyncedAt` update on upsert (`worker/jobs/dbUpsert.ts`) - [ ] T041 [US3] Load test: upsert 500+ policies and measure duration (`scripts/load-tests/upsert-benchmark.js`) - [ ] T042 [US3] Instrument metrics for DB operations (timings, rows inserted/updated) (`worker/logging.ts`) - [ ] T043 [US3] Validate data integrity end-to-end (Graph → transform → DB) (`tests/e2e/full-sync.test.ts`) ## Phase 7: US4 — Frontend Integration & Legacy Cleanup [US4] [X] T044 [US4] Update `lib/actions/policySettings.ts` to remove n8n webhook calls and call `triggerPolicySync()` (`lib/actions/policySettings.ts`) [X] T045 [US4] Update `app/api/policy-settings/route.ts` to be deleted or archive its behavior (`app/api/policy-settings/route.ts`) [X] T046 [US4] Delete `app/api/admin/tenants/route.ts` (n8n polling) (`app/api/admin/tenants/route.ts`) [X] T047 [US4] Remove `POLICY_API_SECRET` and `N8N_SYNC_WEBHOOK_URL` from `.env` and `lib/env.mjs` (`.env`, `lib/env.mjs`) [X] T048 [US4] Grep-check: verify no remaining `n8n` references (repo-wide) (no file) - [ ] T049 [US4] Update docs: remove n8n setup instructions and add worker notes (`docs/worker-deployment.md`) - [ ] T050 [US4] Add migration note to `specs/002-manual-policy-sync/README.md` marking it superseded (`specs/002-manual-policy-sync/README.md`) - [ ] T051 [US4] End-to-end QA: trigger sync from UI and confirm policies saved after cleanup (`tests/e2e/post-cleanup-sync.test.ts`) ## Phase 8: Testing & Validation (no story label) - [ ] T052 Add unit tests for `worker/utils/humanizer.ts` and `policyParser.ts` coverage (`tests/unit/*.test.ts`) - [ ] T053 Add integration tests for worker jobs processing (`tests/integration/worker.test.ts`) - [ ] T054 Run load tests for large tenant (1000+ policies) and record results (`scripts/load-tests/large-tenant.js`) - [ ] T055 Test worker stability (run 1+ hour with multiple jobs) and check memory usage (local script) - [ ] T056 Validate all Success Criteria (SC-001 to SC-008) and document results (`specs/005-backend-arch-pivot/validation.md`) ## Phase 9: Deployment & Documentation (no story label) - [ ] T057 Create `docs/worker-deployment.md` with production steps (`docs/worker-deployment.md`) - [ ] T058 Add deployment config for worker (Dockerfile or PM2 config) (`deploy/worker/Dockerfile`) - [ ] T059 Ensure `REDIS_URL` is set in production Dokploy config and documented (`deploy/README.md`) - [ ] T060 Add monitoring & alerting for worker failures (Sentry / logs / email) (`deploy/monitoring.md`) - [ ] T061 Run canary production sync and verify (`scripts/canary-sync.js`) - [ ] T062 Final cleanup: remove unused n8n-related code paths and feature flags (`grep and code edits`) - [ ] T063 Update `README.md` and `DEPLOYMENT.md` with worker instructions (`README.md`, `DEPLOYMENT.md`) - [ ] T064 Tag release branch `005-backend-arch-pivot` and create PR template (`.github/`) - [ ] T065 Merge PR after review and monitor first production sync (`GitHub workflow`) - [ ] T066 Post-deploy: run post-mortem checklist and close feature ticket (`specs/005-backend-arch-pivot/closure.md`) --- ## Notes - Tasks labeled `[P]` are safe to run in parallel across different files or developers. - Story labels map to spec user stories: `US1` = Manual Sync, `US2` = Graph Fetching, `US3` = Transformation & DB, `US4` = Cleanup & Frontend. - Each task includes a suggested file path to implement work; adjust as needed to match repo layout. # Tasks: Backend Architecture Pivot **Feature**: 005-backend-arch-pivot **Generated**: 2025-12-09 **Total Tasks**: 64 (T001-T066) **Spec**: [spec.md](./spec.md) | **Plan**: [plan.md](./plan.md) --- ## Phase 1: Setup & Infrastructure (8 tasks) **Goal**: Prepare environment, install dependencies, setup Redis and BullMQ queue infrastructure ### Environment Setup - [ ] T001 Install Redis via Docker Compose (add redis service to docker-compose.yml) - [ ] T002 [P] Add REDIS_URL to .env file (REDIS_URL=redis://localhost:6379) - [ ] T003 [P] Update lib/env.mjs - Add REDIS_URL: z.string().url() to server schema - [ ] T004 [P] Update lib/env.mjs - Add REDIS_URL to runtimeEnv object - [ ] T005 Install npm packages: bullmq, ioredis, @azure/identity, tsx ### BullMQ Queue Infrastructure - [X] T006 [P] Create lib/queue/redis.ts - Redis connection wrapper with IORedis - [X] T007 [P] Create lib/queue/syncQueue.ts - BullMQ Queue definition for "intune-sync-queue" - [X] T008 Test Redis connection and queue creation (add dummy job, verify in Redis CLI) --- ## Phase 2: Worker Process Skeleton (6 tasks) **Goal**: Set up worker process entry point and basic job processing infrastructure ### Worker Setup - [ ] T009 Create worker/index.ts - BullMQ Worker entry point with job processor - [ ] T010 [P] Add worker:start script to package.json ("tsx watch worker/index.ts") - [ ] T011 [P] Implement worker event handlers (completed, failed, error) - [ ] T012 [P] Add structured logging for worker events (JSON format) - [ ] T013 Create worker/jobs/syncPolicies.ts - Main sync orchestration function (empty skeleton) - [ ] T014 Test worker starts successfully and listens on intune-sync-queue --- ## Phase 3: Microsoft Graph Integration (9 tasks) **Goal**: Implement Azure AD authentication and Microsoft Graph API data fetching with pagination ### Authentication - [ ] T015 Create worker/jobs/graphAuth.ts - ClientSecretCredential token acquisition - [ ] T016 [P] Implement getGraphAccessToken() using @azure/identity - [ ] T017 Test token acquisition returns valid access token ### Graph API Fetching - [ ] T018 Create worker/jobs/graphFetch.ts - Microsoft Graph API client - [ ] T019 [P] Implement fetchWithPagination() for handling @odata.nextLink - [ ] T020 [P] Create fetchAllPolicies() to fetch from 4 endpoints in parallel - [ ] T021 [P] Add Graph API endpoint constants (deviceConfigurations, compliancePolicies, configurationPolicies, intents) ### Error Handling - [ ] T022 Create worker/utils/retry.ts - Exponential backoff retry logic - [ ] T023 Test Graph API calls with real tenant, verify pagination works for 100+ policies --- ## Phase 4: Data Transformation (11 tasks) **Goal**: Port n8n flattening logic to TypeScript, implement parsers for all policy types ### Policy Parser Core - [ ] T024 Create worker/jobs/policyParser.ts - Main policy parsing router - [ ] T025 [P] Implement detectPolicyType() based on @odata.type - [ ] T026 [P] Implement parsePolicySettings() router function ### Settings Catalog Parser - [ ] T027 Implement parseSettingsCatalog() for #microsoft.graph.deviceManagementConfigurationPolicy - [ ] T028 [P] Implement extractValue() for different value types (simple, choice, group collection) - [ ] T029 Handle nested settings with dot-notation path building ### OMA-URI Parser - [ ] T030 [P] Implement parseOmaUri() for omaSettings[] arrays - [ ] T031 [P] Handle valueType mapping (string, int, boolean) ### Humanizer & Utilities - [ ] T032 Create worker/utils/humanizer.ts - Setting ID humanization - [ ] T033 [P] Implement humanizeSettingId() to remove technical prefixes and format names - [ ] T034 [P] Implement defaultEmptySetting() for policies with no settings ### Validation - [ ] T035 Test parser with sample Graph API responses, verify >95% extraction rate --- ## Phase 5: Database Persistence (7 tasks) **Goal**: Implement Drizzle ORM upsert logic with conflict resolution ### Database Operations - [ ] T036 Create worker/jobs/dbUpsert.ts - Drizzle ORM upsert function - [ ] T037 [P] Implement upsertPolicySettings() with batch insert - [ ] T038 [P] Configure onConflictDoUpdate with policy_settings_upsert_unique constraint - [ ] T039 [P] Update lastSyncedAt timestamp on every sync - [ ] T040 Map FlattenedSetting[] to PolicySetting insert format ### Integration - [ ] T041 Connect syncPolicies() orchestrator: auth → fetch → parse → upsert - [ ] T042 Test full sync with real tenant data, verify database updates correctly --- ## Phase 6: Frontend Integration (4 tasks) **Goal**: Replace n8n webhook with BullMQ job creation in Server Action ### Server Action Update - [ ] T043 Modify lib/actions/policySettings.ts - triggerPolicySync() function - [ ] T044 Remove n8n webhook call (fetch to N8N_SYNC_WEBHOOK_URL) - [ ] T045 Add BullMQ job creation (syncQueue.add('sync-tenant', { tenantId })) - [ ] T046 Test end-to-end: UI click "Sync Now" → job created → worker processes → database updated --- ## Phase 7: Legacy Cleanup (8 tasks) **Goal**: Remove all n8n-related code, files, and environment variables ### File Deletion - [ ] T047 Delete app/api/policy-settings/route.ts (n8n ingestion API) - [ ] T048 Delete app/api/admin/tenants/route.ts (n8n polling API) ### Environment Variable Cleanup - [ ] T049 Remove POLICY_API_SECRET from .env file - [ ] T050 Remove N8N_SYNC_WEBHOOK_URL from .env file - [ ] T051 Remove POLICY_API_SECRET from lib/env.mjs server schema - [ ] T052 Remove N8N_SYNC_WEBHOOK_URL from lib/env.mjs server schema - [ ] T053 Remove POLICY_API_SECRET from lib/env.mjs runtimeEnv - [ ] T054 Remove N8N_SYNC_WEBHOOK_URL from lib/env.mjs runtimeEnv ### Verification - [ ] T055 Run grep search for n8n references: grep -r "POLICY_API_SECRET\|N8N_SYNC_WEBHOOK_URL" --exclude-dir=specs → should be 0 results --- ## Phase 8: Testing & Validation (6 tasks) **Goal**: Comprehensive testing of new architecture ### Unit Tests - [ ] T056 [P] Write unit tests for humanizer.ts - [ ] T057 [P] Write unit tests for retry.ts - [ ] T058 [P] Write unit tests for policyParser.ts ### Integration Tests - [ ] T059 Write integration test for full syncPolicies() flow with mocked Graph API - [ ] T060 Write integration test for database upsert with conflict resolution ### End-to-End Test - [ ] T061 E2E test: Start Redis + Worker, trigger sync from UI, verify database updates --- ## Phase 9: Deployment (5 tasks) **Goal**: Deploy worker process to production environment ### Docker & Infrastructure - [ ] T062 Update docker-compose.yml for production (Redis service with persistence) - [ ] T063 Create Dockerfile for worker process (if separate container) - [ ] T064 Configure worker as background service (PM2, Systemd, or Docker Compose) ### Production Deployment - [ ] T065 Set REDIS_URL in production environment variables - [ ] T066 Deploy worker, monitor logs for first production sync --- ## Dependencies Visualization ``` Phase 1 (Setup) ↓ Phase 2 (Worker Skeleton) ↓ Phase 3 (Graph Integration) ←─┐ ↓ │ Phase 4 (Transformation) ──────┤ ↓ │ Phase 5 (Database) ────────────┘ ↓ Phase 6 (Frontend) ↓ Phase 7 (Cleanup) ↓ Phase 8 (Testing) ↓ Phase 9 (Deployment) ``` **Parallel Opportunities**: - Phase 3 & 4 can overlap (Graph integration while building parsers) - T002-T004 (env var updates) can be done in parallel - T006-T007 (Redis & Queue files) can be done in parallel - T015-T017 (auth) independent from T018-T021 (fetch) - T056-T058 (unit tests) can be done in parallel --- ## Task Details ### T001: Install Redis via Docker Compose **File**: `docker-compose.yml` **Action**: Add Redis service ```yaml services: redis: image: redis:alpine ports: - '6379:6379' volumes: - redis-data:/data restart: unless-stopped volumes: redis-data: ``` **Verification**: `docker-compose up -d redis` && `redis-cli ping` returns PONG --- ### T002-T004: Environment Variable Setup **Files**: `.env`, `lib/env.mjs` **Changes**: 1. Add `REDIS_URL=redis://localhost:6379` to `.env` 2. Add `REDIS_URL: z.string().url()` to server schema 3. Add `REDIS_URL: process.env.REDIS_URL` to runtimeEnv **Verification**: `npm run dev` starts without env validation errors --- ### T005: Install npm Dependencies **Command**: ```bash npm install bullmq ioredis @azure/identity npm install -D tsx ``` **Verification**: Check `package.json` for new dependencies --- ### T006: Create Redis Connection Wrapper **File**: `lib/queue/redis.ts` **Implementation**: See technical-notes.md section "BullMQ Setup" **Exports**: `redisConnection` --- ### T007: Create BullMQ Queue **File**: `lib/queue/syncQueue.ts` **Implementation**: See technical-notes.md section "BullMQ Setup" **Exports**: `syncQueue` --- ### T009: Create Worker Entry Point **File**: `worker/index.ts` **Implementation**: See technical-notes.md section "Worker Implementation" **Features**: - Worker listens on `intune-sync-queue` - Concurrency: 1 (sequential processing) - Event handlers for completed, failed, error --- ### T015-T016: Azure AD Token Acquisition **File**: `worker/jobs/graphAuth.ts` **Implementation**: See technical-notes.md section "Authentication" **Function**: `getGraphAccessToken(): Promise` **Uses**: `@azure/identity` ClientSecretCredential --- ### T018-T021: Graph API Fetching **File**: `worker/jobs/graphFetch.ts` **Functions**: - `fetchWithPagination(url, token): Promise` - `fetchAllPolicies(token): Promise` **Endpoints**: - deviceManagement/deviceConfigurations - deviceManagement/deviceCompliancePolicies - deviceManagement/configurationPolicies - deviceManagement/intents --- ### T024-T034: Policy Parser Implementation **File**: `worker/jobs/policyParser.ts` **Functions**: - `detectPolicyType(odataType: string): string` - `parsePolicySettings(policy: any): FlattenedSetting[]` - `parseSettingsCatalog(policy: any): FlattenedSetting[]` - `parseOmaUri(policy: any): FlattenedSetting[]` - `extractValue(settingInstance: any): any` **Reference**: See technical-notes.md section "Flattening Strategy" --- ### T036-T040: Database Upsert **File**: `worker/jobs/dbUpsert.ts` **Function**: `upsertPolicySettings(tenantId: string, settings: FlattenedSetting[])` **Features**: - Batch insert with Drizzle ORM - Conflict resolution on `policy_settings_upsert_unique` - Update `lastSyncedAt` timestamp **Reference**: See technical-notes.md section "Database Upsert" --- ### T043-T045: Frontend Integration **File**: `lib/actions/policySettings.ts` **Function**: `triggerPolicySync(tenantId: string)` **Before**: ```typescript const response = await fetch(env.N8N_SYNC_WEBHOOK_URL, { method: 'POST', body: JSON.stringify({ tenantId }), }); ``` **After**: ```typescript import { syncQueue } from '@/lib/queue/syncQueue'; const job = await syncQueue.add('sync-tenant', { tenantId, triggeredAt: new Date(), }); return { jobId: job.id }; ``` --- ## Success Criteria Mapping | Task(s) | Success Criterion | |---------|-------------------| | T001-T008 | SC-001: Job creation <200ms | | T041-T042 | SC-002: Sync 50 policies in <30s | | T019-T021 | SC-003: Pagination handles 100+ policies | | T024-T035 | SC-004: >95% setting extraction | | T022-T023 | SC-005: Automatic retry on 429 | | T047-T055 | SC-006: Zero n8n references | | T061, T066 | SC-007: Worker stable 1+ hour | | T041-T042 | SC-008: No data loss on re-sync | --- ## Estimated Effort | Phase | Tasks | Hours | Priority | |-------|-------|-------|----------| | 1. Setup | 8 | 1-2h | P1 | | 2. Worker Skeleton | 6 | 2h | P1 | | 3. Graph Integration | 9 | 4h | P1 | | 4. Transformation | 11 | 6h | P1 | | 5. Database | 7 | 3h | P1 | | 6. Frontend | 4 | 2h | P1 | | 7. Cleanup | 8 | 2h | P1 | | 8. Testing | 6 | 4h | P1 | | 9. Deployment | 5 | 3h | P1 | | **Total** | **64** | **27-29h** | | --- ## Implementation Notes ### Task Execution Order **Sequential Tasks** (blocking): - T001 → T002-T004 → T005 (setup before queue) - T006-T007 → T008 (Redis before queue test) - T009 → T013 (worker before sync skeleton) - T041 → T042 (integration before test) - T043-T045 → T046 (implementation before E2E test) **Parallel Tasks** (can be done simultaneously): - T002, T003, T004 (env var updates) - T006, T007 (Redis + Queue files) - T010, T011, T012 (worker event handlers) - T015-T017, T018-T021 (auth independent from fetch) - T027-T029, T030-T031 (different parser types) - T047, T048 (file deletions) - T049-T054 (env var removals) - T056, T057, T058 (unit tests) ### Common Pitfalls 1. **Redis Connection**: Ensure `maxRetriesPerRequest: null` for BullMQ compatibility 2. **Graph API**: Handle 429 rate limiting with exponential backoff 3. **Pagination**: Always follow `@odata.nextLink` until undefined 4. **Upsert**: Use correct constraint name `policy_settings_upsert_unique` 5. **Worker Deployment**: Don't forget `concurrency: 1` for sequential processing ### Testing Checkpoints - After T008: Redis + Queue working - After T014: Worker starts successfully - After T017: Token acquisition works - After T023: Graph API fetch with pagination works - After T035: Parser extracts >95% of settings - After T042: Full sync updates database - After T046: UI → Worker → DB flow complete - After T055: No n8n references remain - After T061: E2E test passes --- **Task Status**: Ready for Implementation **Next Action**: Start with Phase 1 (T001-T008) - Setup & Infrastructure