tenantpilot/specs/004-policy-explorer-v2/plan.md

# Implementation Plan: Policy Explorer V2

**Branch**: `004-policy-explorer-v2` | **Date**: 2025-12-09 | **Spec**: [spec.md](./spec.md)  
**Input**: Feature specification from `/specs/004-policy-explorer-v2/spec.md`

## Summary

Upgrade the existing Policy Explorer (`/search`) from basic search/table view to advanced data table with:
- **Server-side pagination** (10/25/50/100 rows per page)
- **Multi-column sorting** with ASC/DESC toggle
- **Column management** (show/hide, resize, reorder) persisted in localStorage
- **PolicyType filtering** with multi-select checkboxes
- **Bulk export** (CSV) for selected rows (client-side) and all filtered results (server-side, max 5000)
- **Enhanced detail view** with copy-to-clipboard, raw JSON, and "Open in Intune" link
- **URL state** for shareable filtered/sorted views
- **Sticky header** and compact/comfortable density modes

Technical approach: TanStack Table v8 for client-side table state management, Server Actions for data fetching/export, shadcn/ui primitives for UI consistency, nuqs for URL state, and localStorage for user preferences.

## Technical Context

**Language/Version**: TypeScript 5.x strict mode  
**Primary Dependencies**:  
- Next.js 16+ App Router
- TanStack Table v8 (`@tanstack/react-table`)  
- Drizzle ORM for database queries  
- Shadcn UI components  
- NextAuth.js v4 for tenant isolation  
- URL state: `nuqs` or native `useSearchParams`  
- CSV export: `papaparse` or native string builder  

**Storage**: PostgreSQL (existing `policy_settings` table)  
**Testing**: Jest/Vitest for utils, Playwright for E2E table interactions  
**Target Platform**: Docker containers, modern web browsers (Chrome, Firefox, Safari, Edge)  
**Project Type**: Next.js App Router web application  
**Performance Goals**:  
- Page load: <500ms (50 rows with filters)  
- Sorting/filtering: <200ms  
- CSV export (1000 rows): <2s client-side  
- CSV export (5000 rows): <5s server-side  

**Constraints**:  
- Server-first architecture (all data via Server Actions)  
- No client-side fetching (useEffect + fetch prohibited)  
- TypeScript strict mode (no `any` types)  
- Shadcn UI for all components  
- Azure AD tenant isolation enforced  

**Scale/Scope**: Multi-tenant SaaS, 1000+ policy settings per tenant, 100+ concurrent users

## Constitution Check

*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*

- [X] Uses Next.js App Router with Server Actions (pagination/filtering/export via Server Actions)
- [X] TypeScript strict mode enabled (existing codebase already strict)
- [X] Drizzle ORM for all database operations (policy settings queries use Drizzle)
- [X] Shadcn UI for all new components (Table, Button, Sheet, etc.)
- [X] Azure AD multi-tenant authentication (existing auth, tenant isolation via session)
- [X] Docker deployment with standalone build (existing Dockerfile)

**Result**: ✅ No constitution violations

## Project Structure

### Documentation (this feature)

```text
specs/004-policy-explorer-v2/
├── plan.md              # This file
├── research.md          # Phase 0 output (TanStack Table patterns, CSV strategies)
├── data-model.md        # Phase 1 output (DataTableState, FilterState types)
├── quickstart.md        # Phase 1 output (how to add new columns, filters)
├── contracts/           # Phase 1 output (Server Action signatures)
│   ├── getPolicySettings.yaml
│   ├── exportPolicySettingsCSV.yaml
│   └── types.ts
└── tasks.md             # Phase 2 output (NOT created by this command)
```

### Source Code (repository root)

**Structure Decision**: Next.js App Router structure (existing pattern in codebase)

```text
app/
└── (app)/
    └── search/
        ├── page.tsx                         # Server Component (data fetching)
        ├── PolicyExplorerClient.tsx         # UPDATED: Client Component wrapper
        └── PolicyExplorerTable.tsx          # NEW: TanStack Table component

components/
└── policy-explorer/
    ├── PolicyTable.tsx                      # NEW: Main data table
    ├── PolicyTableColumns.tsx               # NEW: Column definitions
    ├── PolicyTableToolbar.tsx               # NEW: Filters, density toggle, export
    ├── PolicyTablePagination.tsx            # NEW: Pagination controls
    ├── PolicyDetailSheet.tsx                # UPDATED: Add copy buttons, raw JSON
    ├── ColumnVisibilityMenu.tsx             # NEW: Show/hide columns
    └── ExportButton.tsx                     # NEW: CSV export trigger

lib/
├── actions/
│   └── policySettings.ts                    # UPDATED: Add pagination, sorting, export
├── hooks/
│   ├── usePolicyTable.ts                    # NEW: TanStack Table hook
│   ├── useTablePreferences.ts               # NEW: localStorage persistence
│   └── useURLState.ts                       # NEW: URL state sync
└── utils/
    ├── csv-export.ts                        # NEW: Client-side CSV builder
    └── policy-table-helpers.ts              # NEW: Column formatters, sorters

tests/
├── unit/
│   ├── csv-export.test.ts                   # CSV generation logic
│   └── policy-table-helpers.test.ts         # Column utilities
└── e2e/
    └── policy-explorer.spec.ts              # Pagination, sorting, filtering, export
```

## Complexity Tracking

> **No violations** - All constitution checks pass. TanStack Table is a state management library (not a data fetching library), so it complements Server Actions rather than replacing them.

---

## Phase 0: Research & Analysis

### Unknowns to Resolve

1. **TanStack Table Integration Pattern**
   - **Question**: How to integrate TanStack Table with Next.js Server Actions for server-side pagination/sorting?
   - **Research**: Review TanStack Table docs for "manual pagination" mode, check existing patterns in similar Next.js projects
   - **Output**: `research.md` section on TanStack Table + Server Actions integration

2. **CSV Export Strategy**
   - **Question**: When to use client-side vs server-side CSV generation? What's the performance breakpoint?
   - **Research**: Test `papaparse` performance with 100/1000/5000 rows, measure memory usage, compare to server-side stream
   - **Output**: `research.md` section with performance benchmarks and decision matrix

3. **URL State Management**
   - **Question**: Use `nuqs` library or native `useSearchParams` + `useRouter`?
   - **Research**: Compare type safety, SSR compatibility, bundle size
   - **Output**: `research.md` section on URL state library choice

4. **LocalStorage Schema**
   - **Question**: How to version localStorage schema for forward compatibility when adding new features?
   - **Research**: Review best practices for localStorage versioning, schema migration patterns
   - **Output**: `research.md` section on localStorage structure

5. **Sticky Header Implementation**
   - **Question**: Use CSS `position: sticky` or JavaScript scroll listeners? Performance trade-offs?
   - **Research**: Test both approaches with large tables (1000+ rows), measure scroll jank
   - **Output**: `research.md` section on sticky header strategy

### Dependencies & Best Practices

1. **TanStack Table Best Practices**
   - **Task**: Research recommended patterns for server-side pagination, column definitions, type safety
   - **Output**: `research.md` section with code examples

2. **Shadcn Table + TanStack Integration**
   - **Task**: Find examples of shadcn/ui Table primitives used with TanStack Table
   - **Output**: `research.md` section with integration patterns

3. **CSV Escaping & Edge Cases**
   - **Task**: Research proper CSV escaping (commas, quotes, newlines), Excel compatibility
   - **Output**: `research.md` section on CSV generation rules

---

## Phase 1: Design & Contracts

### Data Model Design

**File**: `specs/004-policy-explorer-v2/data-model.md`

#### Entities

**DataTableState** (Client-side ephemeral state)
```typescript
interface DataTableState {
  pagination: {
    pageIndex: number;        // 0-based
    pageSize: 10 | 25 | 50 | 100;
  };
  sorting: Array<{
    id: string;               // column ID
    desc: boolean;            // true = DESC, false = ASC
  }>;
  columnVisibility: {
    [columnId: string]: boolean;
  };
  columnSizing: {
    [columnId: string]: number; // px
  };
  rowSelection: {
    [rowId: string]: boolean;
  };
  density: 'compact' | 'comfortable';
}
```

**FilterState** (Synced with URL + stored in localStorage)
```typescript
interface FilterState {
  policyTypes: string[];      // ['deviceConfiguration', 'compliancePolicy']
  searchQuery: string;         // existing search functionality
  // Future: dateRange, tenant filter (Phase 2)
}
```

**TablePreferences** (Persisted in localStorage)
```typescript
interface TablePreferences {
  version: 1;                  // schema version for migrations
  columnVisibility: { [columnId: string]: boolean };
  columnSizing: { [columnId: string]: number };
  columnOrder: string[];       // ordered column IDs
  density: 'compact' | 'comfortable';
  defaultPageSize: 10 | 25 | 50 | 100;
}
```

#### Database Schema Changes

**None required** - Existing `policy_settings` table has all necessary fields:
- `policyName`, `policyType`, `settingName`, `settingValue`, `graphPolicyId`, `lastSyncedAt`
- Indexes already exist for `tenantId`, `settingName`
- **Recommendation**: Add composite index for sorting performance: `(tenantId, policyType, settingName)`

### API Contracts

**File**: `specs/004-policy-explorer-v2/contracts/`

#### Contract 1: `getPolicySettings`

**File**: `contracts/getPolicySettings.yaml`

```yaml
action: getPolicySettings
description: Fetch policy settings with pagination, sorting, and filtering
method: Server Action

input:
  type: object
  properties:
    page:
      type: number
      minimum: 0
      description: 0-based page index
    pageSize:
      type: number
      enum: [10, 25, 50, 100]
      default: 50
    sortBy:
      type: string
      enum: [settingName, policyName, policyType, lastSyncedAt]
      optional: true
    sortDir:
      type: string
      enum: [asc, desc]
      default: asc
    policyTypes:
      type: array
      items:
        type: string
      optional: true
      description: Filter by policy types
    searchQuery:
      type: string
      optional: true
      description: Text search in settingName/policyName

output:
  type: object
  properties:
    data:
      type: array
      items:
        type: PolicySetting
    meta:
      type: object
      properties:
        totalCount: number
        pageCount: number
        currentPage: number
        pageSize: number
        hasNextPage: boolean
        hasPreviousPage: boolean

errors:
  - UNAUTHORIZED: User not authenticated
  - FORBIDDEN: User not in tenant
  - INVALID_PARAMS: Invalid pagination/sort params
```

#### Contract 2: `exportPolicySettingsCSV`

**File**: `contracts/exportPolicySettingsCSV.yaml`

```yaml
action: exportPolicySettingsCSV
description: Export policy settings as CSV (server-side, max 5000 rows)
method: Server Action

input:
  type: object
  properties:
    policyTypes:
      type: array
      items:
        type: string
      optional: true
    searchQuery:
      type: string
      optional: true
    sortBy:
      type: string
      optional: true
    sortDir:
      type: string
      enum: [asc, desc]
      default: asc
    maxRows:
      type: number
      maximum: 5000
      default: 5000

output:
  type: object
  properties:
    csv:
      type: string
      description: CSV content as string
    filename:
      type: string
      description: Suggested filename (e.g., "policy-settings-2025-12-09.csv")
    rowCount:
      type: number
      description: Number of rows in CSV

errors:
  - UNAUTHORIZED: User not authenticated
  - FORBIDDEN: User not in tenant
  - TOO_MANY_ROWS: Result exceeds maxRows limit
```

### Quickstart Guide

**File**: `specs/004-policy-explorer-v2/quickstart.md`

```markdown
# Policy Explorer V2 - Quickstart

## Adding a New Column

1. Define column in `PolicyTableColumns.tsx`:
   ```typescript
   {
     accessorKey: 'myNewField',
     header: 'My Field',
     cell: ({ row }) => <span>{row.original.myNewField}</span>,
   }
   ```

2. Add to localStorage schema version if changing defaults
3. Update CSV export to include new column

## Adding a New Filter

1. Update `FilterState` type in `lib/types/policy-table.ts`
2. Add UI control in `PolicyTableToolbar.tsx`
3. Update `getPolicySettings` Server Action to handle new filter param
4. Add to URL state in `useURLState.ts`

## Testing Checklist

- [ ] Pagination: Navigate through pages, verify correct data
- [ ] Sorting: Click column headers, verify ASC/DESC toggle
- [ ] Filtering: Select policy types, verify filtered results
- [ ] Column visibility: Hide/show columns, reload page
- [ ] CSV export: Export selected rows, verify content
- [ ] Accessibility: Keyboard navigation, screen reader labels
```

---

## Phase 1.5: Agent Context Update

**Action**: Run `.specify/scripts/bash/update-agent-context.sh copilot`

This script will:
1. Detect AI agent in use (Copilot)
2. Update `.github/copilot-instructions.md`
3. Add new technologies from this plan:
   - TanStack Table v8 for data tables
   - CSV export patterns (client vs server-side)
   - URL state management with nuqs
   - LocalStorage persistence patterns

**Manual additions to preserve**:
- Existing Intune API patterns
- Project-specific conventions

---

## Re-Evaluation: Constitution Check

*Run after Phase 1 design completion*

- [X] TanStack Table uses Server Actions for data (no client-side fetch)
- [X] All types strictly defined (DataTableState, FilterState, contracts)
- [X] Drizzle ORM for queries (no raw SQL)
- [X] Shadcn UI Table primitives (no custom table components)
- [X] Azure AD session enforced in Server Actions
- [X] Docker build unaffected (no new runtime dependencies)

**Result**: ✅ Constitution compliance maintained

---

## Implementation Notes

### Critical Paths

1. **TanStack Table Integration** (Blocking)
   - Must establish pattern for Server Action + TanStack Table manual pagination
   - All other features depend on this foundation

2. **Server Action Updates** (Blocking)
   - `getPolicySettings` must support pagination/sorting before UI can be built

3. **CSV Export** (Parallel after data fetching works)
   - Client-side export can be built independently
   - Server-side export requires Server Action

### Parallel Work Opportunities

- **Phase 1**: Data model design + API contracts can happen in parallel
- **Implementation**: Client-side CSV export + column management can be built while server-side export is in progress
- **Testing**: Unit tests for utils can be written early, E2E tests require full integration

### Risk Mitigation

**Risk**: TanStack Table performance with large datasets (1000+ rows)
- **Mitigation**: Use virtualization (`@tanstack/react-virtual`) if needed
- **Fallback**: Reduce default page size to 25 rows

**Risk**: CSV export memory issues with 5000 rows
- **Mitigation**: Use streaming approach in Server Action
- **Fallback**: Reduce max export to 2500 rows

**Risk**: localStorage quota exceeded (5-10MB limit)
- **Mitigation**: Only store column preferences, not data
- **Fallback**: Clear old preferences, show user warning

---

## Success Metrics

### Performance Benchmarks

- [ ] Page load (50 rows): <500ms
- [ ] Sorting operation: <200ms
- [ ] Filtering operation: <300ms
- [ ] Column resize: <16ms (60fps)
- [ ] CSV export (1000 rows, client): <2s
- [ ] CSV export (5000 rows, server): <5s

### Functional Validation

- [ ] Pagination: All pages load correctly, no duplicate rows
- [ ] Sorting: Correct order for all column types (string, date, number)
- [ ] Filtering: AND logic for multiple filters
- [ ] Column management: Preferences persist across sessions
- [ ] CSV export: Proper escaping, Excel-compatible
- [ ] URL state: Shareable links work correctly
- [ ] Accessibility: Keyboard navigation, ARIA labels

---

## Next Steps

1. **Run Phase 0 Research**:
   ```bash
   # Research TanStack Table patterns
   # Benchmark CSV strategies
   # Choose URL state library
   ```

2. **Generate `research.md`**:
   - Document findings for each unknown
   - Include code examples and performance data

3. **Generate `data-model.md`**:
   - Define TypeScript interfaces
   - Document database schema (no changes)

4. **Generate Contracts**:
   - Create YAML specs for Server Actions
   - Define input/output types

5. **Update Agent Context**:
   ```bash
   .specify/scripts/bash/update-agent-context.sh copilot
   ```

6. **Generate `tasks.md`**:
   ```bash
   # After Phase 1 complete, run:
   /speckit.tasks
   ```

---

**Plan Version**: 1.0  
**Last Updated**: 2025-12-09  
**Status**: Ready for Phase 0 Research