docs/data-loading.md
Linked repository file from this build.
# Data loading and persistence
The production app uses Neon HTTP. A localhost PostgreSQL URL selects the local `pg`
adapter; `docker compose up -d` provides a database bound to loopback. Configure
`DATABASE_URL` with the local credentials in `docker-compose.yml`, then run migrations.
The local adapter executes batches in a transaction, matching the atomic writes used
by the Neon adapter.
## Database targets and migration diagnostics
Keep the development and production credentials together in the ignored `.env` file:
```dotenv
DATABASE_URL=postgresql://...development...
PROD_DATABASE_URL=postgresql://...production...
```
The application continues to use `DATABASE_URL`. The production commands explicitly select
`PROD_DATABASE_URL` and fail if it is missing; they never fall back to development. On the
production deployment itself, its existing `DATABASE_URL` still points to production.
There is no need to swap credentials in your local environment for each migration.
```bash
bun run db:status # Read-only development history and column checks
bun run db:migrate # Apply pending development migrations
bun run db:status:prod # Read-only production history and column checks
bun run db:migrate:prod # Apply pending production migrations
bun run db:backfill:prod # Rebuild every report projection
bun run db:seed:prod --dry-run # Preview the synthetic seed without database calls
bun run db:seed:prod --confirm-production # Replace production review/inbox data
```
`QC_DB_TARGET=production` also selects production for the Drizzle config, seed and report
backfill. `db:seed:prod` uses the same defaults as `db:seed`: `--reset` and 100
September 2026 reviews. It refuses to write without `--confirm-production`. All writes,
including the reset, run in one transaction, so a failed seed changes nothing. It preserves policy tables and accepts the same month, FFY,
count and `--no-reset` options. Use `--dry-run` to inspect the generated dataset without
connecting; seed execution consumes database quota and is never part of migration. Prefer the explicit commands above for routine migration work. `db:push` is
schema synchronization, not a substitute for versioned data migrations.
Production status calls also consume Neon quota. The migration command already checks
history and columns before and after applying changes; an extra status call is optional.
A current database skips migration writes and the duplicate post-check. Backfill is a
separate explicit operation and reads review snapshots in batches; do not run it or
browser/performance suites against production just to check connectivity. Run regression
and failure tests against the disposable local database below.
The migration runner uses PostgreSQL directly and retains Drizzle's existing
`drizzle.__drizzle_migrations` journal. Pending migrations run in Drizzle's transaction.
It takes a session advisory lock to serialize this runner, waits at most 10 seconds for
locks, and limits each statement to five minutes. Neon pooler hostnames are converted to
the corresponding direct hostname for these session settings; the app's runtime pool
configuration is unchanged. See [Neon connection pooling](https://neon.com/docs/connect/connection-pooling).
Each invocation prints the selected target, pending filenames, and statement progress.
Failures include the failing SQL, nested PostgreSQL SQLSTATE, detail, hint and constraint
when available. A redacted JSON report is written to ignored `test-results/database/`
with owner-only permissions. Database URLs and connection passwords are removed; reports
can still contain database row details from PostgreSQL and should stay private.
Status exits with code 1 when migrations are pending, columns are missing, or history is
inconsistent. Success checks column presence, not complete equivalence of indexes,
constraints, types or stored data. Applied-file hash changes are reported as warnings:
Drizzle skips already recorded migrations, so editing old SQL does not update existing
databases. Make subsequent corrections in a new migration. Missing history entries,
unknown timestamps, or existing application tables without a journal stop the runner;
it never invents a baseline or erases history. If history is current but columns are
missing, it reports schema drift instead of claiming a successful no-op migration.
The error `column review_index.review_key does not exist` means the running application
is missing the key column introduced by migration `0006`. The current UUID-based application
also requires `0007`, and `0008` makes the report projection required. Check status on the
intended target. If `0005` is pending, apply it, run the report backfill, then apply the rest.
`0008` fails without changes while any row has no projection.
A failed migration's diagnostic report identifies why it stopped; rerunning a recorded
migration or using `db:push` is not a data-preserving repair strategy.
Test the runner, including failure rollback and old-data upgrades, on a disposable local
database (requires local PostgreSQL and permission to create a database):
```bash
bun test src/lib/db/migrations.test.ts
QC_DB_INTEGRATION=1 bun test src/lib/db/migrations.integration.test.ts
```
The root layout supplies the initial server session. React request caching deduplicates
authentication within a server render; it does not cache actors across requests or users.
Clerk staff roles and scope still come from current server metadata. Browser preferences
cannot grant permissions. The authentication bypass requires a configured test secret,
a matching request secret and a localhost host.
Home, Reviews, Approvals, workspace, Search, Reports and Exports receive their initial
scoped data with the route.
Client filters, pagination and Refresh still use authorized APIs. Initial data is tied to
the actor scope. Home can preload its last measured viewport page size from a bounded,
non-sensitive cookie; the first visit defaults to six rows. A changed viewport can request
a different page size. Signed-out pages do not issue a redundant session request.
Inbox visit writes happen after client navigation, never during server prefetch.
Navigation prefetch is opt-in so visible links do not issue speculative authenticated
database queries. Clerk UI is pinned through `@clerk/ui` because sidebar CSS references
Clerk component structure; update that dependency and the appearance tests together.
## Projections and exports
`review_index.report` stores `{ eligible, variances }`. Snapshot saves update it in the
same transaction as the snapshot and other index fields. Eligibility uses the same
versioned checks as exports, moving repeated validation from reads to writes.
Reports reads projected rows with SQL authorization and filters. Filtered queries also
read distinct authorized examiner/county choices. CSV uses the displayed summary;
pending or failed requests cannot generate stale runs. Inbox and search list projections
exclude the report-only column.
Exports month summaries group eligibility projections by sample month in SQL. Downloads fetch only released snapshots and manager-approved drops in the selected months and staff scope, normalize legacy snapshots,
and recheck all export gates before generating output. They do not load the entire
review history.
Apply the additive migration before deploying code that reads the report projection:
```sh
bun run db:migrate
bun run scripts/backfill-report-index.ts
```
The backfill rebuilds every current projection, 100 snapshots per batch, and guards updates
by snapshot version. It does not modify snapshot history. Reports and export counts read only
the projection, so run it again after a deliberate code change to eligibility for
already-stored rule versions.
Intake reads only scoped review numbers and versions. PDF authorization looks up a matching
storage key within actor scope rather than transferring every current snapshot. Ordinary
quick search uses the index alone; advanced filters and explicit deep search may read JSON
within SQL. The field picker uses a static registry built from current review definitions.
## Refresh and saves
Reviews and Approvals use one folder/list layout, defaulting to 10 slim SQL-paginated
rows, with 5/10/25/50 selectable. They refresh on demand and every five minutes while visible. Paging and sorting reuse cached folder totals; Refresh reloads
them. Home pagers request only the changed list. Opening a review does not refetch sidebar
queues. Home, sidebar queues and the
persistent notification provider refresh every five minutes while visible; notification
focus refreshes only when stale. Explicit preference changes can refresh immediately.
Draft previews copy only modified branches, preserving the saved base and avoiding full
review clones per keystroke. Server execution retains defensive cloning. Field edits
coalesce. Clerk autosave defaults off. When enabled, it saves after one minute without another edit. Blur does not
save. Manual Save, workflow actions and navigation/page hide still flush pending edits.
Computation columns retain explicit Save. Edits made during an autosave wait for the
next autosave; explicit flushes drain them. Failed navigation preserves the editor and
its drafts.
Computation and Review tabs load on first use and then remain mounted to preserve drafts.
Documents and Memo load separately. Loading boundaries keep tab feedback responsive.
## Diagnostics
Catalog, Inbox, Reports and Exports API responses expose `Server-Timing` auth, data and
serialization durations with `private, no-store`. Data timing includes query and
aggregation work; it is not pure database CPU time. `QC_SERVER_TIMING=1` enables server
page auth/data logs containing only route names and durations.
Use the [performance suite](performance-testing.md) to measure production readiness and
transfer. Local tests with synthetic authentication do not establish real Clerk latency,
Vercel compute billing, Neon wire usage or serverless cold-start behavior. Measurements
and historical comparisons belong in `test-results/`, not this document.
Policy migration 0005 creates immutable dated tables, inserts the initial FFY 2026 row, and
invalidates report projections. Run the existing backfill after migration. It does not
rewrite historical reviews or manufacture reporting release flags. Published rows serialize
through a transaction lock so a concurrent older publication cannot follow a newer one.
## Deterministic seeding
Apply migrations through 0008 to a local synthetic database before seeding. Preview a full
year without database access or writes:
```sh
bun run scripts/seed-snap-negative.ts --ffy 2026 --seed 42 --dry-run
```
The default count is 50 closed/suspended and 50 rejected cases per month, or 1,200 per FFY.
Every generated month belongs to an October-dated policy table. FFY 2026 reuses the initial
workbook table. Other years copy those amounts into tables explicitly marked synthetic,
with a fixture threshold of $58 before FFY 2027 and $59 from FFY 2027. These copies are not
verified annual policy. The generator does not change the operational policy defaults.
The reference day defaults to the last day of the final sample month, at noon UTC.
`--as-of YYYY-MM-DD` overrides it. `--month current` deliberately uses the current UTC
month; use an explicit month or FFY for repeatable runs. Generated identifiers, timestamps,
notes, letters, source rows and policy snapshots repeat for the same options.
Current cases cover examiner, supervisor, PEC, manager and CAO stages. Older cases are
released or dropped. Received copies, approval flags and report projections match each
stage. Returned cases include a field-linked correction note. Drops include their kind
and manager approval where complete, and have no PEC approval or memo. Released cases
have a CAO agreement with root cause or no response after the deadline. Memo types are
explicit test fixtures, not an inferred defect map. Where policy response days are unset,
fixtures use a 30-day deadline without publishing that as policy.
Cases use current calculation/check versions, one variance pair at most, a CAO-issued
benefit amount, and farm/non-farm income with profit and loss examples. The existing save
adapter writes queue/search/report projections and applicable initial notifications.
Notification IDs derive from the saved request, recipient and event kind. Seeding does not
create Clerk accounts, user preferences, external tracker records or fictitious PDF files.
`--no-reset` preserves existing reviews in the same sample month and inserts missing ones.
It does not upgrade old snapshots. A review-number collision with another month in the same FFY or a
conflicting published policy stops the command before any reset. Matching policy rows are
reused; missing rows are inserted before reviews. `--reset` clears review and inbox data,
but preserves policy tables. Only use it when replacing the target synthetic dataset.
Generate a multi-year range in one invocation with `--ffy-from` and `--ffy-to`:
```sh
bun run db:seed --ffy-from 2023 --ffy-to 2026 --as-of 2026-09-01
```
Each FFY starts its own number sequences. Closed/suspended cases run from `60001` through
`61999`; rejected cases run from `650001` through `669999`. There are no gaps at `61000`
or `660000`. A batch that exceeds either range within one FFY fails before database writes.
The default seed is **42** when `--seed` is omitted. Each FFY has its own random stream,
so adding other years to a batch does not change its cases when the seed, sample months
and `--as-of` date are the same.
Fixture UUIDs are independent of the random seed: `fixtureReviewKey(reviewNumber, sampleMonth)`
uses UUIDv5 with a fixed namespace and the explicit SNAP business identity. The same
program, fiscal year and review number always produce the same UUID and URL, including
after a reset/reseed. Changing case facts or the random seed does not change the UUID;
changing fiscal year or review number does. Keep the namespace and identity serialization
stable. Tests can use the helper to reference known cases instead of copying UUID literals.
This guarantee concerns identity; repeatable case contents also require the same generator
options and reference date.
Storage and links use an opaque UUID, shared by the snapshot's `id`, the review index,
notifications and touches. New imports receive UUIDv7 on the server. Refreshes retain the
same UUID. The database separately enforces uniqueness on `(program, fiscal_year, review_number)`.
The UI receives fiscal year explicitly; it never parses an ID to find it. Ordinary edits
cannot move a review into another FFY.
Migration 0007 converts existing references together and updates snapshot IDs without deleting
records. It assigns random UUIDs to existing reviews; synthetic seeds use deterministic UUIDv5
only to make fixtures reproducible. Neither form has business meaning to the application.
There are no redirects or lookup fallbacks for earlier number-only or `FFY:number` URLs.
Open existing cases from Search or Reviews after migration.
Deploy the UUID schema and application together: the previous application expects text keys.
This migration rewrites existing identity columns and snapshot JSON once, so schedule it deliberately
on a quota-limited database. It does not seed, rebuild reports, or run on page load.
Opening a missing review returns 404; demo setup must be explicit.
`--fixtures` also writes `policies.json` beside the two attested BIS text files. Those BIS
files represent intake facts; importing them does not reproduce seeded approval history.
## Loading UI
Home, Reviews/Approvals and the review workspace render their destination components with
local placeholder records during navigation. Reports, Exports, Imports, Policy and Search
(Mail desk) also use their real page content for navigation previews. Search results,
export months, catalog refreshes and report charts reuse their actual rows while loading.
`LoadingPreview` masks content, hides placeholders from assistive technology, announces
loading and makes the subtree inert. Its animation respects reduced motion.
Keep geometry in the live components; change placeholder records only when their data shape
changes. Loading views must not mount remote loaders, create review providers, save records
or mark an inbox visit. Home separates its view from fetching; the catalog preview disables
its query and refresh effects. The workspace preview uses a local template without loading
it into the review provider. Export and search previews disable fetch effects, and search
uses the destination query for its filters and columns. Imports and Policy share their
content without mounting another app shell. Routes without a shared preview use a status message.