TenantAtlas/specs/405-json-to-jsonb-data-layer-hardening/plan.md

Implementation Plan: Spec 405 - JSON-to-JSONB Data-layer Hardening

Branch: 405-json-to-jsonb-data-layer-hardening | Date: 2026-06-23 | Spec: specs/405-json-to-jsonb-data-layer-hardening/spec.md Input: Feature specification from specs/405-json-to-jsonb-data-layer-hardening/spec.md

Summary

Prepare and later implement a database-hardening slice that inventories every live PostgreSQL json and jsonb column, classifies all json columns, converts only appropriate queryable trust-layer payload columns to jsonb, adds only query-backed indexes, proves semantic payload preservation, and records local plus staging-like validation. The implementation must not add product behavior, UI surfaces, broad abstractions, normalized replacement tables, or speculative indexes.

Technical Context

Language/Version: PHP 8.4.15, Laravel 12.52.0, Filament 5.2.1, Livewire 4.1.4. Primary Dependencies: Laravel migrations/schema builder, PostgreSQL, Pest 4, Filament/Livewire test helpers where focused browser or rendered-surface proof is needed. Storage: PostgreSQL via Sail/Dokploy; target change is existing column type conversion from json to jsonb where classified CONVERT. Testing: Pest 4, PostgreSQL lane, targeted feature tests, focused browser smoke. Validation Lanes: pgsql, confidence/feature, focused browser, optional profiling/explain for new indexes. Target Platform: TenantPilot Laravel monolith under apps/platform; Spec Kit artifacts under specs/405-json-to-jsonb-data-layer-hardening/. Project Type: Laravel web application plus Spec Kit workflow. Performance Goals: Avoid speculative indexes; each new index must have an existing query path and bounded write-overhead rationale. Constraints: No new product concepts, UI surfaces, Graph calls, authorization model, lifecycle behavior, normalized replacement tables, provider semantics, broad abstractions, or completed-spec rewrites. Scale/Scope: All live PostgreSQL json and jsonb columns; conversion limited to selected existing columns with proof.

Repository Truth And Initial Signals

Laravel Boost confirmed PostgreSQL and current packages:

PHP 8.4.15
Laravel 12.52.0
Filament 5.2.1
Livewire 4.1.4
Pest 4.3.1
PostgreSQL

Laravel 12 migrations support $table->jsonb('column'); implementation should use Laravel migrations where possible and raw PostgreSQL ALTER TABLE ... ALTER COLUMN ... TYPE jsonb USING ...::jsonb where column type alteration requires explicit SQL.

Live PostgreSQL schema inspection on 2026-06-23 found active json columns requiring classification in:

alert_deliveries.payload
alert_rules.tenant_allowlist
audit_logs.metadata
backup_items.assignments
backup_items.metadata
backup_items.payload
backup_schedules.days_of_week
backup_schedules.policy_types
backup_sets.metadata
managed_environment_onboarding_sessions.state
managed_environment_permissions.details
managed_environments.metadata
managed_environments.rbac_canary_results
managed_environments.rbac_last_warnings
policies.metadata
policy_versions.assignments
policy_versions.metadata
policy_versions.scope_tags
policy_versions.secret_fingerprints
policy_versions.snapshot
restore_runs.group_mapping
restore_runs.metadata
restore_runs.preview
restore_runs.requested_items
restore_runs.results
tenant_settings.value
workspace_settings.value

Newer trust-layer paths already use jsonb, including operation_runs.summary_counts, operation_runs.failure_summary, operation_runs.context, baseline scope_jsonb/summary_jsonb/meta_jsonb, evidence summaries, findings evidence, review pack summaries/options, stored report payloads, provider connection metadata/scopes, and review publication resolution payloads.

Technical Approach

1. Record branch, HEAD, dirty state, and git diff --check.
2. Query PostgreSQL information_schema.columns for all json and jsonb columns and collect row counts, null counts, defaults, indexes, and constraints.
3. Map each column to model casts, factories/fixtures, query usages, Filament/rendered usage, tests, and sensitive-data boundaries.
4. Classify each column as CONVERT, KEEP_JSON, ALREADY_JSONB, DEPRECATED, or DECISION_REQUIRED.
5. Write one or more focused migrations converting only CONVERT columns.
6. Add only query-backed JSONB indexes with explicit proof and rollback/drop strategy.
7. Update casts or query code only if required by tests after conversion.
8. Add PostgreSQL/feature tests proving type conversion, semantic preservation, model read/write behavior, scope/authorization boundaries, and representative domain regressions.
9. Run focused browser proof over existing payload-backed surfaces.
10. Produce implementation-report.md with the inventory matrix, validation results, remaining findings, staging status, and final gate result.

Likely Affected Repository Surfaces

Preparation identifies likely inspection and implementation surfaces; implementation must verify exact paths before editing.

apps/platform/database/migrations/
apps/platform/app/Models/
apps/platform/database/factories/
apps/platform/tests/
apps/platform/app/Support/
apps/platform/app/Services/
apps/platform/app/Jobs/
apps/platform/app/Filament/
apps/platform/resources/views/
specs/405-json-to-jsonb-data-layer-hardening/implementation-report.md

Likely model/cast inspection targets include:

App\Models\AlertDelivery
App\Models\AlertRule
App\Models\AuditLog
App\Models\BackupItem
App\Models\BackupSchedule
App\Models\BackupSet
App\Models\ManagedEnvironment
App\Models\ManagedEnvironmentOnboardingSession
App\Models\ManagedEnvironmentPermission
App\Models\Policy
App\Models\PolicyVersion
App\Models\RestoreRun
App\Models\TenantSetting
App\Models\WorkspaceSetting

UI / Surface Guardrail Plan

• Guardrail scope: backend data-layer hardening with focused rendered regression proof.
• Affected routes/pages/actions/states/navigation/panel/provider surfaces: none changed by implementation unless the spec is updated first.
• No-impact class: backend-only schema/storage conversion.
• Native vs custom classification summary: N/A.
• Shared-family relevance: evidence/report viewers, provider readiness, OperationRun proof, backup/restore proof, and audit metadata are regression consumers only.
• State layers in scope: persistence and existing rendered state regression only.
• Audience modes in scope: operator/MSP and customer/read-only only for regression proof; no disclosure behavior changes.
• Decision/diagnostic/raw hierarchy plan: unchanged; raw payloads remain technical/audit detail.
• Raw/support gating plan: unchanged.
• One-primary-action / duplicate-truth control: unchanged.
• Handling modes by drift class or surface: report-only unless conversion causes a confirmed regression, then minimal in-scope fix.
• Repository-signal treatment: review-mandatory for changed database/runtime paths.
• Special surface test profiles: backend storage conversion plus focused browser proof.
• Required tests or manual smoke: PostgreSQL migration tests, feature regressions, focused browser smoke.
• Exception path and spread control: any required UI edit is out of scope and must stop implementation for spec/plan update.
• Active feature PR close-out entry: Guardrail / database hardening.
• UI/Productization coverage decision: No UI surface impact.
• Coverage artifacts to update: none unless implementation unexpectedly changes rendered UI; then stop and update spec/plan first.
• No-impact rationale: storage type conversion and proof only.
• Navigation / Filament provider-panel handling: no panel/provider changes.
• Screenshot or page-report need: focused browser proof may produce screenshots/logs as evidence; no page report required without UI changes.

Product Surface Contract Plan

• Product Surface Contract reference: docs/product/standards/product-surface-contract.md as regression lens only.
• No-legacy posture: canonical replacement; no compatibility shims or dual-write paths.
• Page archetype and surface budget plan: N/A for changed code; browser proof names existing inspected archetypes.
• Technical Annex and deep-link demotion plan: unchanged; conversion must not expose raw payloads, internal IDs, OperationRun links, or evidence deep links by default.
• Canonical status vocabulary plan: unchanged.
• Product Surface exceptions: none.
• Browser verification plan: focused existing-surface regression proof required.
• Human Product Sanity plan: final implementation report confirms unchanged trust semantics.
• Visible complexity outcome target: neutral.
• Implementation report target: specs/405-json-to-jsonb-data-layer-hardening/implementation-report.md.

Filament / Livewire / Deployment Posture

• Livewire v4 compliance: Livewire 4.1.4 confirmed; no Livewire code change planned.
• Panel provider registration location: Laravel 12 panel providers remain in apps/platform/bootstrap/providers.php; no panel change.
• Global search posture: no resource global search posture changed. If implementation touches a resource unexpectedly, verify View/Edit/global search safety or stop for spec update.
• Destructive/high-impact action posture: no actions added or changed. Existing destructive/high-impact action proof from Specs 401-404 must not regress.
• Asset strategy: no assets, no FilamentAsset registration, no new filament:assets requirement beyond existing deployment baseline.
• Testing plan: database/migration tests, feature/domain regressions, focused browser proof for existing payload-backed surfaces.
• Deployment impact: migrations and staging/Dokploy validation. No env vars, queues, scheduler, storage, assets, routes, provider scopes, or panel providers planned.

Shared Pattern & System Fit

• Cross-cutting feature marker: yes at data storage level.
• Systems touched: database schema, existing casts, existing query paths, tests, implementation report.
• Shared abstractions reused: Laravel migrations, existing models/casts, existing scoped query paths, existing tests/browser fixtures.
• New abstraction introduced? why?: none planned.
• Why the existing abstraction was sufficient or insufficient: existing models and services already own payload meaning; storage conversion does not require a new runtime layer.
• Bounded deviation / spread control: any new helper must be justified as a narrow test/support helper, not a runtime framework.

OperationRun UX Impact

• Touches OperationRun start/completion/link UX?: no.
• Central contract reused: N/A.
• Delegated UX behaviors: N/A.
• Surface-owned behavior kept local: N/A.
• Queued DB-notification policy: N/A.
• Terminal notification path: N/A.
• Exception path: none.

Provider Boundary & Portability Fit

• Shared provider/platform boundary touched?: storage inspection only.
• Provider-owned seams: provider raw/permission/readiness payload keys remain unchanged.
• Platform-core seams: storage type, workspace/managed-environment scope, audit/report/evidence ownership, migration safety.
• Neutral platform terms / contracts preserved: workspace, managed environment, provider, connection, operation, evidence, report, backup, restore.
• Retained provider-specific semantics and why: existing Microsoft/Intune payload keys remain provider-owned payload content.
• Bounded extraction or follow-up path: none for this spec.

Domain / Model Implications

• No new model, table, persisted artifact, enum, status family, route, action, provider type, or source of truth is introduced.
• Existing model casts may remain unchanged because Laravel treats json and jsonb similarly at the application layer; implementation must update casts only if tests prove current behavior breaks.
• Existing raw payload keys and product meaning remain unchanged.
• Columns with unclear ownership or product semantics must be classified DECISION_REQUIRED, not converted by assumption.

Data / Migration Implications

Migration pattern for direct conversions:

ALTER TABLE table_name
ALTER COLUMN column_name TYPE jsonb
USING column_name::jsonb;

Rollback pattern where feasible:

ALTER TABLE table_name
ALTER COLUMN column_name TYPE json
USING column_name::json;

Implementation must preserve:

• nullable state
• defaults
• constraints
• existing indexes unless intentionally replaced
• row counts
• application read/write behavior

Implementation must document rollback limitations:

• jsonb normalizes key order
• duplicate JSON object keys may be normalized
• semantic content must remain preserved, but textual representation may differ

Index Strategy

Allowed only with existing query proof:

• GIN index for existing containment/query usage.
• Expression index for existing frequently queried JSONB key.
• Partial index for existing filtered JSONB key access.

Rejected:

• index every converted column
• index because jsonb supports it
• index for future lifecycle/export/dashboard guesses
• index without write-overhead note

Each index requires a row in the implementation report:

Index | Table/Column | Query Path | Reason | Expected Benefit | Write Overhead Risk | Proof

RBAC / Security / Audit Implications

• No RBAC behavior changes.
• Any changed query involving payload keys must retain existing workspace and managed-environment scoping.
• Non-member and wrong-scope access must remain deny-as-not-found where applicable.
• Customer-safe report/review/evidence paths must not expose raw payloads because of conversion or debug output.
• Reports, logs, screenshots, and test fixtures must not include secrets, tokens, raw credential payloads, sensitive raw provider payloads, or customer-sensitive raw payloads.
• Audit metadata conversion must preserve actor/context fields.

OperationRun / Evidence / Result Truth Implications

The plan distinguishes:

• Execution truth: existing OperationRun status/outcome/summary/context; live columns are already jsonb, but regression proof still checks rendering.
• Artifact truth: ReviewPack, StoredReport, evidence snapshots/items, backup sets/items.
• Backup/snapshot truth: policy versions, backup payloads, baseline/evidence payloads.
• Recovery/evidence truth: restore previews/results, evidence currentness, report receipts.
• Operator next action: unchanged existing UI states and actions.

Test Strategy

Required test groups:

1. PostgreSQL schema/type tests for every converted column.
2. Migration semantic preservation tests for representative non-sensitive payloads.
3. Model cast/read-write tests where converted columns are written by Eloquent models.
4. Query-path tests for any changed JSON key query and any new index.
5. Evidence/currentness regression tests for converted evidence/report/review payloads.
6. OperationRun/audit regression tests where summary/context/audit metadata is in scope.
7. Provider readiness/freshness/permission regression tests where provider/environment payloads are converted.
8. Backup/restore payload regression tests for backup items/sets/schedules and restore preview/results.
9. Review/report receipt regression tests for review pack/stored report/customer output.
10. Authorization/scope tests for changed payload queries.
11. Focused browser smoke for representative existing payload-backed pages.

Preferred validation commands:

cd apps/platform && ./vendor/bin/sail php vendor/bin/pest -c phpunit.pgsql.xml --filter=Spec405
cd apps/platform && ./vendor/bin/sail artisan test --filter=Evidence
cd apps/platform && ./vendor/bin/sail artisan test --filter=OperationRun
cd apps/platform && ./vendor/bin/sail artisan test --filter=Provider
cd apps/platform && ./vendor/bin/sail artisan test --filter=Backup
cd apps/platform && ./vendor/bin/sail artisan test --filter=Restore
cd apps/platform && ./vendor/bin/sail artisan test --filter=ReviewPack
cd apps/platform && ./vendor/bin/sail artisan test --filter=StoredReport

Use narrower commands where implementation creates focused Spec 405 tests.

Rollout And Deployment Considerations

• Local: run through Laravel Sail against PostgreSQL.
• Staging: validate migration execution, app boot, queue/browser relevant proof, representative pages, and rollback/forward notes where safe.
• Production: do not claim production readiness unless staging-like validation passes or the final report explicitly records PASS WITH CONDITIONS.
• Migrations: assess table size/row count and lock risk before direct conversion.
• Env vars: none expected.
• Queues/scheduler/storage/assets: none expected, but affected flows may rely on existing workers/storage and must not regress.
• Dokploy: database migration and app boot validation required before production promotion.

Risk Controls

• Stop before migration if inventory is incomplete.
• Stop before conversion if a high-risk column has unclear product ownership.
• Use semantic JSON comparisons, not raw string comparisons.
• Classify large-table conversion risk before direct ALTER COLUMN.
• Do not add speculative indexes.
• Do not print sensitive payload samples.
• Do not rewrite completed specs.
• If a conversion causes UI/rendered regression, fix the data-layer cause if bounded; otherwise stop for spec update.

Constitution Check

• Inventory-first: PASS; inventory and source-of-truth classification precede conversion.
• Read/write separation: PASS; no Graph/write product behavior is added.
• Graph contract path: PASS; no Graph calls are added.
• Deterministic capabilities: N/A; no capability resolver changes.
• RBAC-UX: PASS; authorization must not change and changed payload queries require scope tests.
• Workspace isolation: PASS; scoped query proof required.
• Tenant/managed-environment isolation: PASS; managed-environment scope remains enforced.
• Run observability: N/A; no new OperationRun.
• OperationRun start UX: N/A.
• Ops-UX lifecycle/summary counts: no changes to lifecycle; current operation_runs JSONB posture is inspected.
• Data minimization: PASS; sensitive payloads are not dumped.
• Test governance: PASS; PostgreSQL and focused browser proof are explicit.
• Proportionality: PASS; storage conversion only, no new product truth.
• No premature abstraction: PASS; no new framework.
• Persisted truth: PASS; no new persisted entity/table.
• Behavioral state: PASS; no new state.
• UI semantics: PASS; no UI semantics added.
• Shared pattern first: PASS; existing models/services/tests are reused.
• Provider boundary: PASS; provider payload keys unchanged.
• V1 explicitness / few layers: PASS.
• Spec discipline / bloat check: PASS; one coherent data-layer hardening package.
• Product Surface Contract Gate: PASS as no rendered surface change plus focused regression proof.

Test Governance Check

• Test purpose / classification by changed surface: PostgreSQL migration/type proof, feature/domain regression, focused browser smoke.
• Affected validation lanes: pgsql, confidence, browser.
• Why this lane mix is the narrowest sufficient proof: storage type is PostgreSQL-specific and payload-backed pages require rendered regression proof, but no full browser audit is needed.
• Narrowest proving command(s): focused Spec 405 pgsql/feature/browser tests once created.
• Fixture / helper / factory / seed / context cost risks: minimal explicit fixtures only; no shared default broadening.
• Expensive defaults or shared helper growth introduced?: no planned.
• Heavy-family additions, promotions, or visibility changes: focused browser proof only.
• Surface-class relief / special coverage rule: no UI code changes; browser is regression proof.
• Closing validation and reviewer handoff: verify inventory coverage, migration proof, no sensitive payload output, no speculative indexes, and final gate result.
• Budget / baseline / trend follow-up: record if pgsql/browser runtime materially increases.
• Review-stop questions: Is every json column classified? Is every conversion justified? Does every new index have existing query proof? Did any UI or product behavior change? Was staging-like validation completed or properly conditioned?
• Escalation path: document-in-feature for bounded findings; follow-up-spec for unsafe conversion, online migration need, or unresolved product/schema decision.
• Active feature PR close-out entry: Guardrail / Database Hardening.
• Why no dedicated follow-up spec is needed: the slice is bounded unless implementation finds large-table online migration or unresolved product/schema ownership.

Implementation Phases

Phase 1 - Inventory and Classification

Build schema inventory from PostgreSQL, migrations, models, casts, factories, query paths, and tests. Produce the draft inventory matrix before any migration.

Phase 2 - Conversion and Index Design

Write focused migration(s) only for CONVERT columns. Add no index unless tied to an existing query path. Keep rollback explicit.

Phase 3 - Regression and Preservation Proof

Add PostgreSQL and feature tests proving column type, semantic preservation, read/write behavior, scope/authorization, and representative domain behavior.

Phase 4 - Browser, Staging, and Report

Run focused browser proof, staging-like validation where available, final dirty-state checks, and complete the implementation report with gate result.

Complexity Tracking

Violation	Why Needed	Simpler Alternative Rejected Because
None planned	N/A	N/A

Proportionality Review

• Current operator problem: trust-layer payload storage is inconsistent and can undermine future evidence/report/lifecycle work.
• Existing structure is insufficient because: older json payload columns conflict with current PostgreSQL/jsonb posture and query/index expectations.
• Narrowest correct implementation: classify all columns, convert only selected existing columns, add only query-backed indexes, and prove behavior.
• Ownership cost created: migration review, PostgreSQL tests, focused browser proof, and implementation report.
• Alternative intentionally rejected: blanket conversion plus blanket indexes.
• Release truth: current-release data-layer readiness.