TenantAtlas/specs/405-json-to-jsonb-data-layer-hardening/spec.md

# Feature Specification: Spec 405 - JSON-to-JSONB Data-layer Hardening

**Feature Branch**: `405-json-to-jsonb-data-layer-hardening`
**Created**: 2026-06-23
**Status**: Draft / Ready for implementation preparation review
**Type**: Database hardening / migration proof / data integrity spec
**Runtime posture**: Data-layer conversion and proof only. No new product behavior, no new product concepts, and no new rendered UI surfaces.
**Input**: User-provided "Spec 405 - JSON-to-JSONB Data-layer Hardening" draft from `/Users/ahmeddarrazi/.codex/attachments/b7628b1e-f536-40b3-8364-0f6f476c59ac/pasted-text.txt`, current repo truth, Spec 400 product-contract audit context, Specs 401-404 implementation/proof context, roadmap/spec-candidate queue, and live PostgreSQL schema inspection through Laravel Boost.

## Candidate Selection Context

- **Selected candidate**: JSON-to-JSONB Data-layer Hardening.
- **Source location**: Direct user-provided Spec 405 draft, promoted from the Spec 400 P1 data-layer concern that important structured payload columns still use PostgreSQL `json` even though the project standard prefers `jsonb` for queryable policy, evidence, backup, restore, report, provider, and audit payloads.
- **Why selected**: `docs/product/spec-candidates.md` currently reports no safe automatic next-best-prep target, but the operator supplied a concrete manual candidate. The live PostgreSQL schema confirms active `json` columns remain in trust-layer tables such as `policy_versions`, `backup_items`, `restore_runs`, `audit_logs`, `alert_deliveries`, `managed_environment_permissions`, settings tables, and provider/environment metadata paths, while newer foundations already use `jsonb`.
- **Roadmap relationship**: Supports current governance and architecture hardening by reducing data-layer ambiguity before governance artifact lifecycle/retention, report-output expansion, or broader runtime/browser audit work proceeds.
- **Close alternatives deferred**:
  - `governance-artifact-lifecycle-retention-runtime`: broader P2 lifecycle semantics; must not be hidden inside a storage-type conversion.
  - `provider-readiness-onboarding-productization`: optional UX/productization work; unrelated to payload storage type proof.
  - `cross-domain-indicator-runtime-follow-through`: semantic runtime adoption; not a database hardening gate.
  - `manual-system-panel-browser-fixture-or-audit-procedure`: validation procedure work; not a payload storage hardening slice.
  - Reopening Specs 400-404 or Specs 378-404: forbidden. They are read-only context and proof lineage only.
- **Completed-spec guardrail result**:
  - No `specs/405-json-to-jsonb-data-layer-hardening/` package existed before this preparation.
  - A local and remote branch named `405-dach-trust-datenschutz-security-website-surface` existed before preparation, but no TenantPilot `specs/405-*` package existed. The operator explicitly supplied Spec ID 405, so this package uses the requested number and records the branch-prefix collision as preparation context.
  - Specs 400, 401, 402, 403, and 404 contain audit, implementation-report, validation, browser, or completed-task signals and are read-only historical context. Their close-out notes, validation results, completed task markers, screenshots, smoke history, and implementation reports must not be rewritten, normalized, unchecked, or removed.
- **Smallest viable implementation slice**: Inventory every PostgreSQL `json` and `jsonb` column, classify every `json` column, convert only appropriate queryable trust-layer `json` columns to `jsonb`, add only query-backed indexes, prove semantic payload preservation and existing behavior, run PostgreSQL migration/rollback validation, run focused regression tests and representative browser proof for payload-backed surfaces, and record a final JSON/JSONB inventory matrix plus gate result.
- **Feature description for Spec Kit**: Prove and harden TenantPilot structured payload storage by inventorying all PostgreSQL `json` and `jsonb` columns, converting only appropriate queryable trust-layer `json` columns to `jsonb`, preserving payload semantics and scope boundaries, and documenting local plus staging-like validation without adding new product behavior or UI surfaces.

## Spec Candidate Check *(mandatory - SPEC-GATE-001)*

- **Problem**: TenantPilot stores critical structured proof and operational metadata in a mix of PostgreSQL `json` and `jsonb`; important older payload columns are still `json` even though the project standard prefers `jsonb` for queryable snapshots, backup/restore payloads, evidence/report metadata, and audit context.
- **Today's failure**: Future governance, evidence, report, backup, restore, and lifecycle work can build on inconsistent storage assumptions, miss query/index opportunities, or let agents guess which payloads are queryable/trust-layer data versus low-risk configuration values.
- **User-visible improvement**: Operators and reviewers get stronger confidence that evidence, backup, restore, report, provider, audit, and OperationRun-adjacent payloads remain semantically preserved while becoming query-ready where the product already depends on structured payload proof.
- **Smallest enterprise-capable version**: A schema inventory, explicit classification matrix, targeted `json` to `jsonb` migrations with rollback notes, query-backed indexes only, semantic preservation tests, PostgreSQL lane validation, focused browser proof for representative payload-backed surfaces, and a spec-local implementation report.
- **Explicit non-goals**: No new evidence semantics, provider state model, backup/restore feature, governance lifecycle/retention/export/delete/hold behavior, report template, UI surface, navigation, authorization model, broad normalization, event-sourcing redesign, or completed-spec rewrite.
- **Permanent complexity imported**: One spec package, one future implementation report/inventory matrix, targeted migrations, focused tests, and possibly justified indexes. No new persisted entity/table, enum/status family, runtime abstraction, product vocabulary, UI framework, or provider framework is approved.
- **Why now**: Spec 400 identified the data-layer gap as a P1 condition, Specs 401-404 closed adjacent action, authorization, evidence/currentness, and management-report runtime proof gates, and the live schema still shows older `json` payload columns in trust-layer areas.
- **Why not local**: Converting one column locally would not prove all payload columns were classified, excluded columns were intentional, indexes were query-backed, and report/evidence/backup/restore/provider/audit behavior remained unchanged.
- **Approval class**: Core Enterprise.
- **Red flags triggered**: Foundation/hardening language and many tables. Defense: the slice is bounded to existing storage types, existing payload semantics, and proof; it forbids new models, product concepts, broad frameworks, speculative indexes, and UI changes.
- **Score**: Nutzen: 2 | Dringlichkeit: 2 | Scope: 2 | Komplexitaet: 1 | Produktnaehe: 1 | Wiederverwendung: 2 | **Gesamt: 10/12**
- **Decision**: approve as a bounded data-layer hardening and proof package.

## Problem Statement

TenantPilot depends on structured payloads for policy snapshots, backup items, restore previews/results, audit metadata, provider permission/readiness details, evidence summaries, report receipts, review packs, settings, alerts, and OperationRun-adjacent proof. Some of these payloads remain PostgreSQL `json`, while newer code already uses `jsonb` for similar trust-layer data.

This spec answers:

```text
Can TenantPilot safely convert appropriate important json payload columns to jsonb without data loss, product behavior changes, authorization drift, evidence/currentness drift, report/restore/backup regressions, or speculative indexing?
```

A conversion is successful only when every `json` column was intentionally classified, selected conversions preserve semantic payload content, excluded columns have explicit reasons, migrations and rollback behavior are documented, PostgreSQL validation passes, and existing trust-layer behavior remains unchanged.

## Product / Business Value

- Reduces future data-layer ambiguity around queryable governance, evidence, backup, restore, provider, report, and audit payloads.
- Makes later lifecycle/retention/export/reporting work smaller and safer by establishing clear payload storage posture first.
- Prevents false confidence from application casts alone; storage type, query paths, indexes, tests, and staging-like validation must agree.
- Keeps trust-layer payload semantics stable while hardening the substrate.

## Primary Users / Operators

- Product owner and release reviewer deciding whether the platform is ready for lifecycle/retention and customer-output expansion.
- Engineering reviewer validating migration safety, rollback posture, and test coverage.
- MSP/operator relying on backup, restore, evidence, provider, report, and audit proof.
- Support/platform operator diagnosing payload-backed behavior without raw sensitive payload leakage.

## Spec Scope Fields *(mandatory)*

- **Scope**: database schema inventory, classification, migrations, indexes, tests, validation, focused browser proof, and final data-layer implementation report.
- **Primary Routes / Surfaces**: No route or UI surface is added or changed. Focused browser proof later inspects representative existing payload-backed surfaces only, such as Evidence Overview, Monitoring -> Operations, Provider readiness/detail, Backup/Restore proof, Review Pack or Stored Report output.
- **Data Ownership**:
  - Workspace-owned payloads remain workspace scoped.
  - Managed-environment-owned payloads remain workspace plus managed-environment scoped.
  - Customer-safe report/review payloads remain bound to existing `ReviewPack`, `StoredReport`, `EnvironmentReview`, and evidence ownership rules.
  - No new persisted entity/table/artifact is introduced.
- **RBAC**:
  - No authorization behavior changes are allowed.
  - Existing policies, gates, customer-output gates, deny-as-not-found behavior, and global search posture must continue to pass.
  - Any payload query changed during implementation must preserve workspace and managed-environment scoping.

For canonical-view or mixed-scope surfaces:

- **Default filter behavior when environment-context is active**: unchanged. Payload storage conversion must not alter route-owned workspace/environment context or explicit page filters.
- **Explicit entitlement checks preventing cross-tenant leakage**: any changed JSON key query must retain existing scoped query predicates before or with the payload predicate; tests must cover changed high-risk query paths.

## No Legacy / No Backward Compatibility Constraint *(mandatory)*

TenantPilot is pre-production for this data-layer hardening.

- **Compatibility posture**: canonical storage hardening with current payload semantics preserved.
- **Legacy aliases, fallback readers, hidden routes, duplicate UI, old labels, or historical fixtures kept?**: no new compatibility aliases, dual-write paths, fallback readers, hidden routes, duplicate UI, or legacy fixtures are allowed unless this spec is updated with an explicit exception.
- **Why clean replacement is safe now**: PostgreSQL `json` columns already contain valid JSON. Conversion to `jsonb` is a storage hardening change, and semantic behavior must be proven through decoded JSON equality/key checks rather than raw string order.

## UI Surface Impact *(mandatory - UI-COV-001)*

Does this spec add, remove, rename, or materially change any reachable UI surface?

- [x] No UI surface impact
- [ ] Existing page changed
- [ ] New page/route added
- [ ] Navigation changed
- [ ] Filament panel/provider surface changed
- [ ] New modal/drawer/wizard/action added
- [ ] New table/form/state added
- [ ] Customer-facing surface changed
- [ ] Dangerous action changed
- [ ] Status/evidence/review presentation changed
- [ ] Workspace/environment context presentation changed

## UI/Productization Coverage

N/A - no reachable UI surface impact.

- **No-impact rationale**: This spec changes only storage type, migrations, proof, and tests. It may require focused browser proof to show existing payload-backed pages still render correctly after conversion, but it must not edit Filament resources/pages/widgets, Livewire components, Blade views, CSS, JavaScript, navigation, routes, modals, actions, tables, forms, or rendered copy.

## Product Surface Impact *(mandatory for UI-affecting specs; otherwise N/A)*

Reference: `docs/product/standards/product-surface-contract.md`.

- **Product Surface Contract applies?**: no rendered product surface is changed. The contract is used only as a regression lens because payload conversion supports evidence, provider, restore, backup, report, and OperationRun proof surfaces.
- **Page archetype**: N/A for implementation changes. Browser proof later names inspected existing archetypes.
- **Primary user question**: N/A for runtime UI; regression question is "Do existing payload-backed surfaces still render truthful state after conversion?"
- **Primary action**: N/A.
- **Surface budget result**: N/A.
- **Technical Annex / deep-link demotion**: unchanged. Raw payloads, IDs, OperationRun links, evidence deep links, source keys, and provider payloads must not become more visible.
- **Canonical status vocabulary**: unchanged.
- **Visible complexity impact**: neutral.
- **Product Surface exceptions**: none.

## Browser Verification Plan *(mandatory)*

- **Browser proof required?**: yes during implementation, as backend regression proof for existing payload-backed surfaces.
- **No-browser rationale**: N/A for implementation because conversion affects storage used by existing rendered trust surfaces, even though no UI file is changed.
- **Focused path when required**:
  1. Evidence overview or evidence anchor surface renders the expected current/stale/missing proof state.
  2. Monitoring -> Operations or OperationRun detail renders summary/context/failure proof correctly.
  3. Provider detail/readiness/freshness surface renders provider metadata/permission state correctly.
  4. Backup or restore proof surface renders preview/result/payload-backed state correctly.
  5. Review Pack, Stored Report, or report receipt surface renders evidence/report payload state correctly.
- **Primary interaction to execute**: read existing pages, inspect payload-backed status and links, verify no 500, Livewire, Filament, console, or network errors, and verify no raw/internal payload exposure.
- **Console, Livewire, Filament, network, and 500-error checks**: required for focused browser proof.
- **Full-suite failure triage**: unrelated browser failures may be documented only when the focused Spec 405 path is green and evidence supports the classification.

## Human Product Sanity Check *(mandatory)*

- **Required?**: yes for final implementation report, focused on unchanged trust semantics rather than new UI design.
- **No-human-sanity rationale**: no new rendered surface, but human sanity must confirm the change did not make existing payload-backed surfaces claim stronger proof, expose raw data, or confuse current/released/failed/partial states.
- **Reviewer questions**: Are payload semantics unchanged? Are proof/currentness states still truthful? Are technical details still demoted? Are authorization/customer-safe boundaries unchanged? Does the final report avoid overclaiming staging/production readiness?
- **Planned result location**: `specs/405-json-to-jsonb-data-layer-hardening/implementation-report.md`.

## Product Surface Merge Gate Checklist *(mandatory)*

- [x] No-legacy posture or approved exception recorded.
- [x] Product Surface Impact is `N/A - no rendered product surface changed` with rationale.
- [x] Browser proof is planned as focused regression proof for existing payload-backed surfaces.
- [x] Human Product Sanity is planned for unchanged trust semantics.
- [x] Product Surface exceptions are `none`.
- [x] Implementation report will state tests/browser result, Livewire v4 compliance, provider registration location, global search posture, destructive/high-impact action posture, asset strategy, deployment impact, visible complexity outcome, and no completed-spec rewrite assertion.

## Cross-Cutting / Shared Pattern Reuse

- **Cross-cutting feature?**: yes at the data layer; no runtime interaction family is implemented.
- **Interaction class(es)**: evidence/report viewers, OperationRun proof, provider readiness, backup/restore proof, audit metadata, alert delivery payloads, and settings payloads are storage consumers only.
- **Systems touched**: database migrations, model casts only where required, query paths only where required, tests, and implementation report.
- **Existing pattern(s) to extend**: Laravel migrations, PostgreSQL information schema inspection, existing model casts, existing scoped query paths, existing Pest PostgreSQL lane, existing browser smoke patterns.
- **Shared contract / presenter / builder / renderer to reuse**: N/A - no UI/runtime interaction contract is added.
- **Why the existing shared path is sufficient or insufficient**: Existing payload consumers are sufficient; the spec hardens storage beneath them. Any missing query path or cast defect must be fixed locally, not by creating a new abstraction.
- **Allowed deviation and why**: none planned.
- **Consistency impact**: payload meaning, scope, authorization, report/evidence truth, and audit metadata must stay consistent before and after conversion.
- **Review focus**: no speculative indexes, no data loss, no raw payload leakage, no broad normalization, no customer-output semantics change, and no completed-spec rewrite.

## OperationRun UX Impact

- **Touches OperationRun start/completion/link UX?**: no UX change. Existing `operation_runs` columns are already `jsonb` in the live schema, but OperationRun-adjacent proof paths must be inspected and regression-tested.
- **Shared OperationRun UX contract/layer reused**: N/A.
- **Delegated start/completion UX behaviors**: N/A.
- **Local surface-owned behavior that remains**: N/A.
- **Queued DB-notification policy**: N/A.
- **Terminal notification path**: N/A.
- **Exception required?**: none.

## Provider Boundary / Platform Core Check

- **Shared provider/platform boundary touched?**: yes at storage inspection level only.
- **Boundary classification**: mixed storage consumers; provider-specific payload semantics remain provider-owned, while storage and scope rules are platform-core.
- **Seams affected**: provider connection metadata, managed-environment permission details, policy snapshots/versions, backup/restore payloads, report/evidence payloads, and audit metadata.
- **Neutral platform terms preserved or introduced**: payload, metadata, evidence, operation, workspace, managed environment, provider connection, report, backup, restore.
- **Provider-specific semantics retained and why**: Microsoft/Intune keys inside existing payloads remain unchanged because the spec does not rename or reinterpret payload keys.
- **Why this does not deepen provider coupling accidentally**: storage type conversion does not introduce provider-specific platform contracts, labels, or taxonomies.
- **Follow-up path**: broader provider readiness/productization remains a separate manual candidate if later evidence requires it.

## UI / Surface Guardrail Impact

N/A - no operator-facing surface change. Focused browser proof later verifies existing surfaces still render after backend conversion.

## Decision-First Surface Role

N/A - no surface changed.

## Audience-Aware Disclosure

N/A - no disclosure changed. Tests and browser proof must still confirm raw/internal payloads do not become default-visible or customer-visible because of conversion or query changes.

## UI/UX Surface Classification

N/A - no surface changed.

## Operator Surface Contract

N/A - no new or materially refactored operator-facing page.

## Proportionality Review *(mandatory when structural complexity is introduced)*

- **New source of truth?**: no.
- **New persisted entity/table/artifact?**: no new entity/table. Migrations may alter storage type of existing columns only.
- **New abstraction?**: no.
- **New enum/state/reason family?**: no.
- **New cross-domain UI framework/taxonomy?**: no.
- **Current operator problem**: trust-layer payload storage is inconsistent and makes later evidence/report/lifecycle work riskier.
- **Existing structure is insufficient because**: leaving older high-risk payloads as `json` conflicts with the repo's PostgreSQL/jsonb posture and makes query/index proof ambiguous.
- **Narrowest correct implementation**: classify all JSON columns, convert only selected existing columns, add only query-backed indexes, and prove semantic preservation.
- **Ownership cost**: migration review, rollback notes, PostgreSQL lane tests, focused regression/browser proof, and final inventory/report maintenance inside the feature.
- **Alternative intentionally rejected**: blanket conversion of every `json` column and blanket GIN indexes. Both would import unnecessary migration risk and write overhead without product proof.
- **Release truth**: current-release data-layer hardening required before broader lifecycle/reporting work.

## Testing / Lane / Runtime Impact *(mandatory for runtime behavior changes)*

- **Test purpose / classification**: Feature, PostgreSQL, and Browser regression. Unit tests only for pure helpers if implementation introduces a bounded helper for semantic comparison or schema inspection.
- **Validation lane(s)**: PostgreSQL lane for migrations/jsonb/indexes; confidence/feature lane for payload behavior; focused browser lane for rendered regression proof.
- **Why this classification and these lanes are sufficient**: SQLite cannot prove PostgreSQL `jsonb`, GIN/expression indexes, or type conversion. Feature tests prove Laravel casts and behavior; browser proof proves representative existing pages still render.
- **New or expanded test families**: one focused Spec 405 test set only; avoid broad browser audit and heavy governance expansion.
- **Fixture / helper cost impact**: minimal explicit fixtures for selected converted columns; no global seed/default context broadening.
- **Heavy-family visibility / justification**: browser proof is explicit because payload storage backs critical trust surfaces; it must stay focused and not become a full UI audit.
- **Special surface test profile**: backend data-layer hardening with existing rendered trust-surface smoke.
- **Standard-native relief or required special coverage**: no Filament component changes; browser proof is regression-only.
- **Reviewer handoff**: verify every converted column appears in the inventory matrix, every excluded `json` column has a reason, every new index has query proof, and no test or fixture leaks sensitive payloads.
- **Budget / baseline / trend impact**: possible PostgreSQL lane runtime increase; document if material.
- **Escalation needed**: document-in-feature unless a conversion is unsafe or requires product/schema decision, then follow-up-spec.
- **Active feature PR close-out entry**: Guardrail / database hardening.
- **Planned validation commands**:
  - `git status --short --branch`
  - `git diff --check`
  - `cd apps/platform && ./vendor/bin/sail artisan migrate`
  - `cd apps/platform && ./vendor/bin/sail artisan migrate:rollback --step=1` only in a disposable local/test database when safe
  - `cd apps/platform && ./vendor/bin/sail php vendor/bin/pest -c phpunit.pgsql.xml --filter=Spec405`
  - targeted feature tests for evidence, OperationRun, provider, backup, restore, review/report payload behavior
  - focused browser smoke for representative payload-backed surfaces

## User Scenarios & Testing *(mandatory)*

### User Story 1 - Inventory and classify structured payload columns (Priority: P1)

As an engineering reviewer, I need a complete inventory and classification of all `json` and `jsonb` columns so I can verify that storage decisions are intentional rather than guessed from column names.

**Why this priority**: No migration should be written until every `json` column has a reasoned decision.

**Independent Test**: Run the information-schema query and compare it to the implementation report matrix; every live `json` or `jsonb` column appears once with action, reason, risk, model/cast, query usage, row count, null/default details, and validation requirement.

**Acceptance Scenarios**:

1. **Given** the live PostgreSQL schema contains `json` and `jsonb` columns, **When** the implementation report is reviewed, **Then** every column is present with one of `CONVERT`, `KEEP_JSON`, `ALREADY_JSONB`, `DEPRECATED`, or `DECISION_REQUIRED`.
2. **Given** a high-risk payload column remains `json`, **When** the report is reviewed, **Then** it has an explicit reason and risk classification rather than being omitted.

### User Story 2 - Convert selected trust-layer payloads safely (Priority: P1)

As an operator and release reviewer, I need selected important payload columns converted to `jsonb` without semantic data loss so evidence, backup, restore, provider, report, and audit behavior remains trustworthy.

**Why this priority**: Conversion is the core hardening value and must be proven before follow-up lifecycle or reporting work depends on queryable payloads.

**Independent Test**: Run the migration against PostgreSQL, compare semantic JSON content before/after for selected samples, verify column types, null/default preservation, and rollback notes.

**Acceptance Scenarios**:

1. **Given** selected columns contain representative payloads, **When** the migration runs, **Then** decoded JSON content and required keys are preserved after conversion to `jsonb`.
2. **Given** a column has nulls or a default, **When** the migration runs, **Then** nullable/default behavior remains unchanged unless the implementation report records a spec-approved reason.
3. **Given** a large or unclear table cannot be safely converted directly, **When** classification is completed, **Then** the column is marked `DECISION_REQUIRED`; the final gate may be `PASS WITH CONDITIONS`, but the column is not forced through a speculative migration.

### User Story 3 - Prove existing behavior and report readiness honestly (Priority: P2)

As a product owner, I need a final implementation report that proves application behavior did not regress and clearly states whether data-layer readiness is `PASS`, `PASS WITH CONDITIONS`, or `FAIL`.

**Why this priority**: Storage conversion is not complete unless runtime behavior, authorization/scope boundaries, and representative rendered surfaces still work.

**Independent Test**: Run targeted feature/PostgreSQL tests, focused browser proof, and final dirty-state checks; verify the implementation report records commands, results, remaining findings, and staging-like validation status.

**Acceptance Scenarios**:

1. **Given** converted payloads support evidence/provider/backup/restore/report/audit flows, **When** targeted tests and browser proof run, **Then** existing behavior remains unchanged and no raw/internal payload becomes customer-visible.
2. **Given** staging/Dokploy validation is unavailable, **When** the final report is written, **Then** production data-layer readiness is no stronger than `PASS WITH CONDITIONS`.

### Edge Cases

- `jsonb` normalizes object key order; tests must compare decoded semantic content, not raw JSON strings.
- Duplicate JSON object keys cannot be reconstructed after `jsonb` normalization; rollback notes must record this limitation.
- Null payloads must remain null where null was the prior contract.
- Empty object/array defaults must remain unchanged where defaults already exist.
- Columns with no model, no current query usage, or unclear product ownership must not be converted silently.
- Indexes must not be added for hypothetical future dashboards, lifecycle work, or export queries.
- Sensitive payload samples must not be printed in reports, logs, screenshots, or committed fixtures.
- Changed JSON key queries must preserve workspace/managed-environment scoping before or with payload predicates.

## Requirements *(mandatory)*

### Functional Requirements

- **FR-405-001**: Implementation MUST inventory every PostgreSQL `json` and `jsonb` column from the live schema before writing migrations.
- **FR-405-002**: The inventory matrix MUST include table, column, current type, row count, nullable state, default, model, payload purpose, existing queries, existing casts, recommended action, reason, and risk.
- **FR-405-003**: Every `json` column MUST be classified as `CONVERT`, `KEEP_JSON`, `DEPRECATED`, or `DECISION_REQUIRED`; every `jsonb` column MUST be classified as `ALREADY_JSONB`.
- **FR-405-004**: Implementation MUST convert only columns classified `CONVERT`.
- **FR-405-005**: High-risk trust-layer payloads MUST be converted or explicitly justified, including policy snapshots/metadata, backup payloads, restore preview/results/metadata, audit metadata, provider permission/readiness details, report/review/evidence payloads, alert delivery payloads, and relevant governance/finding metadata.
- **FR-405-006**: Implementation MUST preserve payload semantic content using decoded JSON equality, key-existence checks, required field checks, or domain-specific state assertions.
- **FR-405-007**: Implementation MUST preserve nullability, defaults, constraints, row counts, and application write/read behavior unless the spec is updated with an explicit exception.
- **FR-405-008**: Implementation MUST document rollback behavior and limitations, including `jsonb` key-order normalization and duplicate-key normalization.
- **FR-405-009**: Implementation MUST add indexes only for existing query paths and MUST document table/column/key, query path, reason, expected benefit, write-overhead risk, and validation proof.
- **FR-405-010**: Implementation MUST NOT add speculative GIN, expression, or partial indexes for future dashboards, lifecycle work, export work, or "might need later" reasoning.
- **FR-405-011**: Implementation MUST update model casts only where required to preserve existing Laravel behavior after conversion.
- **FR-405-012**: Implementation MUST fix code paths that assumed textual JSON representation only when tests prove the assumption breaks after `jsonb` conversion.
- **FR-405-013**: Implementation MUST not rename payload keys or change payload meaning.
- **FR-405-014**: Implementation MUST not add new product features, UI surfaces, navigation, authorization models, provider semantics, governance lifecycle behavior, report templates, or normalized replacement tables.
- **FR-405-015**: Implementation MUST prove evidence/currentness behavior from Spec 403 does not regress for converted evidence/report/review payloads.
- **FR-405-016**: Implementation MUST prove authorization and customer-safe boundaries from Specs 401 and 402 do not regress for any changed payload query path.
- **FR-405-017**: Implementation MUST prove management-report PDF/runtime behavior from Spec 404 does not regress where stored report/review payloads or report metadata are in scope.
- **FR-405-018**: Implementation MUST run PostgreSQL-specific migration/type assertions for converted columns.
- **FR-405-019**: Implementation MUST run targeted read/write tests for converted payloads.
- **FR-405-020**: Implementation MUST run focused browser proof for representative payload-backed surfaces.
- **FR-405-021**: Implementation MUST produce `specs/405-json-to-jsonb-data-layer-hardening/implementation-report.md` using the required report structure.
- **FR-405-022**: Implementation MUST record final gate result as `PASS`, `PASS WITH CONDITIONS`, or `FAIL`.
- **FR-405-023**: Implementation MUST not claim production data-layer readiness if staging-like PostgreSQL validation is unavailable or incomplete.
- **FR-405-024**: Implementation MUST preserve completed historical spec artifacts as read-only context.

### Non-Functional Requirements

- **NFR-405-001**: Migrations MUST be safe, incremental, reversible where practical, and explicit about lock/runtime risk.
- **NFR-405-002**: PostgreSQL-specific behavior MUST be validated in PostgreSQL, not SQLite.
- **NFR-405-003**: Reports, logs, screenshots, and test output MUST avoid secrets, tokens, raw credential payloads, and sensitive raw provider/customer payloads.
- **NFR-405-004**: The implementation MUST avoid widening test fixtures, browser setup, workspace/member context, or provider setup beyond the narrow proof needs.
- **NFR-405-005**: Any remaining P0/P1 data-layer proof gap MUST be reported, not silently deferred.

## Key Entities *(include if feature involves data)*

- **JSON/JSONB Column Inventory Row**: Spec-local report row representing a live schema column, its current type, ownership, usage, action, reason, and risk.
- **Conversion Migration**: One or more Laravel migrations that alter selected existing columns from `json` to `jsonb` using PostgreSQL-safe conversion.
- **Index Justification Row**: Spec-local report row proving a new JSONB index is tied to an existing query path and bounded overhead.
- **Data Validation Row**: Spec-local report row proving row counts, null counts, type changes, and semantic payload checks before and after conversion.

## Success Criteria *(mandatory)*

### Measurable Outcomes

- **SC-405-001**: 100% of live PostgreSQL `json` and `jsonb` columns appear in the inventory matrix.
- **SC-405-002**: 100% of `json` columns have an explicit action and reason.
- **SC-405-003**: 100% of converted columns have type assertions proving `jsonb` after migration.
- **SC-405-004**: 100% of converted columns have semantic preservation proof for representative non-sensitive sample payloads or an explicit no-row/no-sample note.
- **SC-405-005**: 100% of added JSONB indexes have existing query-path justification and validation proof.
- **SC-405-006**: Targeted evidence, OperationRun, provider, backup, restore, review/report, authorization/scope, and customer-safe regression tests pass or produce documented bounded findings.
- **SC-405-007**: Focused browser proof covers at least five representative payload-backed surfaces or documents exact unavailable paths and resulting gate condition.
- **SC-405-008**: Final implementation report records gate result, validation commands, staging-like validation status, remaining findings, and recommended next step.

## Initial Repo Truth Snapshot

Live PostgreSQL schema inspection on 2026-06-23 found these active `json` columns requiring classification:

```text
alert_deliveries.payload
alert_rules.tenant_allowlist
audit_logs.metadata
backup_items.assignments
backup_items.metadata
backup_items.payload
backup_schedules.days_of_week
backup_schedules.policy_types
backup_sets.metadata
managed_environment_onboarding_sessions.state
managed_environment_permissions.details
managed_environments.metadata
managed_environments.rbac_canary_results
managed_environments.rbac_last_warnings
policies.metadata
policy_versions.assignments
policy_versions.metadata
policy_versions.scope_tags
policy_versions.secret_fingerprints
policy_versions.snapshot
restore_runs.group_mapping
restore_runs.metadata
restore_runs.preview
restore_runs.requested_items
restore_runs.results
tenant_settings.value
workspace_settings.value
```

The same inspection found several newer trust-layer columns already using `jsonb`, including baseline, evidence, finding, review pack, review publication resolution, provider connection, stored report, support request, notification, and operation run payload columns. Implementation must still verify casts and query paths for `ALREADY_JSONB` rows where relevant.

## Required Final Report Structure

Implementation MUST create `specs/405-json-to-jsonb-data-layer-hardening/implementation-report.md` with these sections:

1. Candidate Gate Result.
2. Scope Confirmation.
3. Dirty State.
4. JSON/JSONB Inventory Matrix.
5. Migrations Added.
6. Index Justification.
7. Runtime Changes Made.
8. Tests Added or Updated.
9. Data Validation.
10. Browser Proof.
11. Regression Proof.
12. Staging Validation.
13. Remaining Findings.
14. Deferred Items.
15. Validation Commands.
16. Product Surface, Filament, Deployment, and Recommended Next Step Close-Out, including tests/browser result, Livewire v4 compliance, provider registration location, global search posture, destructive/high-impact action posture, asset strategy, deployment impact, visible complexity outcome, no completed-spec rewrite assertion, and recommended next step.

## Gate Rules

- **PASS**: no P0/P1 findings remain; high-risk `json` columns are converted or justified; local and staging-like PostgreSQL validation pass; payload semantic preservation, targeted tests, and representative browser proof pass.
- **PASS WITH CONDITIONS**: no P0 remains; local/test migration proof is strong; remaining P1 conditions such as staging validation or rollback proof are bounded and documented.
- **FAIL**: any P0 remains; payload data is lost; migration cannot safely run; application behavior regresses; evidence/currentness or authorization/customer-safe boundaries regress; or high-risk `json` columns remain unconverted without justification.

## Follow-up Spec Candidates

- Governance Artifact Lifecycle & Retention runtime, only after data-layer readiness is `PASS` or conditions do not affect lifecycle storage.
- Bounded online migration strategy if any large table cannot safely use direct `ALTER COLUMN ... TYPE jsonb`.
- Provider readiness/onboarding productization if browser/regression proof reveals provider-facing user friction unrelated to storage type.
- Full browser/runtime audit remains separate and must not be folded into this spec.

## Assumptions

- PostgreSQL is the authoritative local/staging database for this proof.
- The product is still pre-production under the constitution, so compatibility shims are not required unless explicitly approved.
- Existing payload keys and product meanings are correct unless tests reveal a defect; this spec does not reinterpret payload schemas.
- Staging/Dokploy validation may be externally blocked; if so, final readiness must be conditional.

## Open Questions

- None blocking preparation. Implementation must decide column actions from live schema inventory and repo usage evidence, not from this preparation draft alone.