Story 524: OCR fallback for scanned documents

Why this exists (the catastrophe it prevents). A huge fraction of legal documents are scans — a photographed order, a faxed affidavit, a printed-then-rescanned filing. They are PDFs or images with no text layer: native extraction (Story-523) finds nothing. Without an OCR fallback those documents reach the chronology engine as blank pages — present in the register, useless for search or review. Story 524 is the fallback that makes scanned documents first-class.

Where OCR sits in the flow

native = LiteParse.extract(bytes, content_type)        # Story-523
if native.stripped_text_length >= NATIVE_TEXT_MIN_CHARS:
    ... # born-digital: done, no OCR
else:
    ocr = OcrPort.recognize(bytes, content_type)       # ◀── Story-524: the fallback
    persist_ocr_result(firm_id, document_id, ocr)      # engine = textract
    low_confidence = any page below its engine threshold

The decisions that matter

1. Reuse the merged OcrPort — do not write a second Textract integration

OCR is reached through the OcrPort seam from Story-1500: recognize(document_bytes, content_type) -> OcrResult, with the Textract adapter as the only implementation today (PaddleOCR is a future config swap, not a rewrite). The worker depends only on the neutral OcrResult — it never learns Textract's response shape, its 0–100 confidence scale, or its geometry. The port owns all of that.

2. One Textract call per document — never per page

The fallback makes a single OcrPort.recognize call for the document and persists every page it returns. Combined with the worker's idempotency (a redelivered, already-terminal document is skipped; a crashed attempt commits nothing), this guarantees the property #532 asked for: Textract is never re-paid per page, and never re-paid on redelivery.

3. Confidence is engine-relative; the gate is per engine

A page is "low confidence" only relative to its own engine's threshold (confidence_threshold(ocr.engine) — Textract 0.80, Paddle 0.60), read from the engine that travelled with the result, never hardcoded. If any page is below its gate, the document moves to needs_review so a human looks — but every recognized page still persists as succeeded, so a single fuzzy page never blocks the rest.

4. Persisted through the single page writer

OCR pages land in documents.document_pages via persist_ocr_result — the same single writer the native path uses (persist_native_result). Each row carries engine='textract', engine_version, the neutral 0–1 block geometry as JSONB, and mean_confidence; the upsert key (firm_id, document_id, page_number) makes re-OCR idempotent. The concatenated OCR text feeds the same documents.search_vector tsvector as the native path, so scanned documents are full-text searchable from day one.

5. Same scan gate, same trace, same audit

The OCR branch inherits every guarantee of the worker: it only runs on stored (virus-scanned) bytes; each transition appends to document_status_events under the continuous trace_id; and a document.extracted audit row records page count, engine, and outcome (never the extracted text — the document audit subject excludes ocr_text/extracted_text as PII).

Verification

The integration suite drives the OCR branch against a real DB: a non-PDF / text-layer-less input routes to OCR, a high-confidence result indexes with a textract page row, and a low-confidence result (a page below the Textract threshold) routes the document to needs_review while the page itself stays succeeded.

One sentence to remember: when native extraction finds no text, the worker makes one call through the OcrPort, persists the neutral result, and flags the document — not its pages — for review when confidence is low.