Story 1855: the import ledger — making “100% of your files were captured” a falsifiable claim

Why this exists (the catastrophe it prevents). A lawyer drags a 400-file case folder into PRAMAAN. We tell them “100% accounted for.” But today there is no server-side record of what they dropped. The browser enumerates the folder, presigns some files, PUTs some of those, and we count what arrived. The files the browser never enumerated (a permission-denied subfolder, a symlink, an over-cap file) and the ones we presigned but the PUT never completed are in neither the numerator nor the denominator. So “100%” is captured / captured — a tautology that is always true and means nothing. If three privileged exhibits silently fell out of a 400-file drop, we would still report 100%, and a court would later find the gap. This Story persists the denominator so the headline becomes accounted / expected, with every dropped file enumerated and reason-coded rather than invisible.

🛰 Satellite view: a third lifecycle axis, as a new table — never a new status value

documents.documents already carries two strictly-orthogonal lifecycle axes: processing (pending→uploaded→stored→…) and retention (live/deleted). The single most load-bearing design decision (design note §0) is that import/transport accountability is a third axis and lives in its own tables, never as an overloaded value on an existing column. The reason: a folder batch, an expected-vs-observed count, and a pre-document materialisation failure simply cannot be modelled on a row that, by definition, only exists once a file has already become a document. The denominator has to live somewhere that can hold a file that never became a document at all.

So this Story creates two tables in the documents schema:

It also widens the documents processing_status CHECK to add the unreadable terminal — not used yet, but this Story owns all V1 import schema (single migration, single alembic head), so the readability story (T6) can build on it with zero migrations of its own.

✈️ Airplane view: where the denominator is minted, and when each line settles

The denominator is minted at selection — the moment the browser enumerates the picked folder, before any presign. That is the whole point: if you mint the count later (at presign, or at upload) you have already lost the files that never got that far.

  SELECTION                PRESIGN            FINALIZE           MALWARE-SCAN SETTLE
  (mint expected_count)    (URL issued)       (S3 PUT seen)      (clean/threat/unscannable)
       |                       |                  |                       |
  expected ----presign----> presigned --land--> landed -----settle-----> accounted   (numerator++)
       |                       |
       | (never presigned)     | (presigned, no PUT)
       v                       v
   abandoned <--import close-- orphaned                     rejected (pre-document failure,
   (un-selected tail)          (TTL sweep)                   reason-coded: over-cap, corrupt...)

The happy path is expected → presigned → landed → accounted, wired into the existing upload pipeline at exactly the three seams the AC names:

🔬 Microscopic view: the decisions that matter

1. “Accounted” is the scan-settle, because import accountability ⊥ processing

A file is accounted-for the moment it has crossed the malware-scan gate — whether it came out clean (stored), quarantined (failed), or parked for triage (needs_review). All three are a real, scanned, managed document; none is a silent drop. We deliberately do not tie accounted to downstream extraction reaching indexed: that would couple the import axis to the processing axis the design's own V-AXIS invariant says must stay orthogonal, and it would wedge an import “open” for as long as extraction runs. The scan-settle is the single accounting point this Story owns; when the extraction worker (T6) later reaches its own terminals it builds on this same ledger, it does not redefine it.

2. The counter is O(1) — the query count never scales with file count (DR-1/DR-2)

Each settle issues exactly one UPDATE document_imports SET accounted_count = accounted_count + 1. We never recount items per row. The structural backstop is a table CHECK, accounted_count <= expected_count: a double-count bug trips a constraint violation instead of silently reporting >100%. The completeness read is two bounded queries (one aggregate over imports, one GROUP BY (state, reason_code) over items) — it loops over states (a fixed set), never over rows.

3. The V-LEDGER invariant — the falsifiable replacement for the vibes headline

When an import is closed:

  expected_count == count(items WHERE state = 'accounted')
                  + count(items WHERE state IN ('abandoned','rejected'))   -- honestly dropped
  AND no item is left in a non-terminal state.

Every input file lands in exactly one terminal bucket: accounted (made it in), abandoned (the un-uploaded tail of a closed session), or rejected (a named pre-document failure). Closing is refused while any item is still landed (mid-scan) — you cannot honestly declare the manifest reconciled while a crate is still on the forklift. This invariant is pinned by an executable test that runs the real routes and the real scan-settle against a real, empty first-run firm.

4. Firm + matter isolation, RLS written by hand

Both tables enable + force RLS with the firm-isolation + staff-with-grant policies copied verbatim from documents.documents (autogenerate cannot see RLS, so it is hand-written in the migration). Matter visibility is enforced one level up: every entry point resolves the matter through business.matters, whose RLS runs business.can_read_matter — so a caller who cannot see the matter gets a 404 before any ledger row is touched. The app role may read/insert/update (the state machine + the counter) but never delete: the ledger is a durable accountability record.

5. Observability: one registry-approved span at the ledger boundary (pramaan-docs ADR-124)

The three domain-operation boundaries — mint, settle, close — each open one DOCUMENT_IMPORT_LEDGER_TRANSITION manual span via the shared manual_span helper (no raw tracer calls; the name is a new entry in the closed ADR-124 registry). Attributes are low-cardinality and non-PII only: import.stage, the expected/accounted counts, the import business key — never a filename, path, or content. The settle leg runs inside apply_scan_result's upload_traceparent-rehydrated context (Bug-1852), so one trace_id stays continuous browser → finalize → malware-scan → settle; we extend that trace, we do not re-break it. This Story adds no new queue, so there is no new producer/consumer boundary and no queue-depth metric to own — the ZIP-expand queue (T7/C1) is where that lands.

Verification

All three new tests pass; the existing documents, malware-scan, and manual-span suites stay green (53 passed) against make test-db — the ledger wiring did not disturb presign/finalize/scan. Ruff is clean on every changed file (the repo's pre-existing lint debt is untouched); git diff --check is clean; openapi/app-api.json is regenerated and committed, admin-api.json is unchanged. Consumer mirror: the new app routes + DTOs need a pnpm codegen in pramaan-app to mirror the contract.

One sentence to remember: the import ledger persists the denominator — every selected file becomes one ledger line that must end accounted, abandoned, or rejected — so “100%” finally means accounted / expected with every dropped file named, never a silent loss the headline hid.