Story 1584: the native-extraction adapter — LiteParse default, pypdfium2 fallback, paragraph bbox on both paths

Why this exists (the catastrophe it prevents). Story-523 gave the worker native extraction, but it was hardwired to one engine that the code called "LiteParse" while actually being plain pypdfium2 — text only, no geometry. Two problems compound from there. (1) The name was a lie: the real LiteParse wheel — run-llama's Rust core that bundles the same PDFium plus per-item bounding boxes, fonts, and confidence — was never installed; the codebase squatted its name on a lesser engine, so adopting the real thing would collide. (2) Born-digital pages had no boxes: only OCR pages got document_pages.blocks, so the source-highlight UI ("jump to where this claim appears on page 4") could highlight a scanned page but not a born-digital one. Story-1584 fixes both: native extraction goes behind a port with two swappable vendors (the real LiteParse as default, pypdfium2 as a fallback), and a single vendor-neutral composition step gives both paths paragraph-level boxes.

🛰️ Satellite: the port, the two vendors, the one composition step

document bytes
   │
   ├─ NativeExtractorPort  (DOCUMENT_NATIVE_EXTRACTOR selects the vendor)
   │     ├─ LiteParseAdapter  (default)  ── real wheel, ocr_enabled=False
   │     │      → page text + per-ITEM blocks (bbox 0..1 + confidence)
   │     └─ PdfiumAdapter    (fallback)  ── pypdfium2, text only, no blocks
   │
   │  born-digital? (stripped_text_length ≥ NATIVE_TEXT_MIN_CHARS)
   │     yes ─▶ persist_native_result ─┐
   │     no  ─▶ OcrPort/Textract ──────┤   ← scanned/no-text-layer pages only
   │            → page text + per-LINE blocks
   │                                    │
   │                  compose_paragraphs(line/item blocks)   ← ONE step, both paths
   │                  → paragraph blocks: union bbox, mean confidence, reading order
   │                                    │
   └────────────────────────────────────▶ documents.document_pages.blocks (uniform JSONB)

The key shape: each vendor and Textract emit line/item-grained blocks; the same compose_paragraphs runs over them at persistence to emit paragraph blocks. So "paragraph box per page" is one function, written once, identical for LiteParse and Textract — not two parallel implementations that drift.

✈️ Airplane: the files

🔬 Microscopic: the decisions that matter

1. A port, because there is now more than one native engine

The whole reason OcrPort exists is that nothing downstream may depend on which engine produced the text. Native extraction now has the same property — LiteParse and pypdfium2 — so it gets the same treatment. The worker holds a NativeExtractorPort; swapping vendors is a one-env change (DOCUMENT_NATIVE_EXTRACTOR=pdfium), never a rewrite. build_native_extractor rejects an unknown vendor loudly at cold start, so a typo fails fast — never a silent wrong-vendor parse.

2. Native-only is the safety contract, and it makes traineddata unnecessary

The extraction worker is VPC-attached for RDS with no internet egress and a read-only root filesystem. LiteParse's built-in Tesseract defaults to downloading eng.traineddata from GitHub into $HOME on first OCR — both the download (no NAT → socket hang) and the write (read-only rootfs) would fail in prod while passing every local test (the classic "fixtures lie / first real run breaks" trap). We sidestep it entirely by constructing LiteParse(ocr_enabled=False): a text-less page returns empty text, Tesseract never initializes, no download is attempted, and the worker routes that page to the OcrPort (Textract) instead. So no eng.traineddata is bundled — it is unreachable by construction. This was verified empirically against the real wheel: parsing a glyph-less PDF with ocr_enabled=False produces zero text and never creates the ~/.tesseract-rs cache dir.

3. Engine stays pdfium; the vendor lives in engine_version — no migration

LiteParse bundles the same PDFium text layer pypdfium2 wraps, so the persisted document_pages.engine enum stays pdfium for both native vendors — the existing CHECK constraint already admits it, and this Story ships no migration. The vendor is distinguished by engine_version: liteparse/2.1.1 vs pypdfium2/5.10.1, so a page row is still auditable to the exact engine that produced it.

4. Paragraph bbox: composition, not pricier Textract LAYOUT

Both engines capture geometry at the line/item grain, but the UI wants a box per paragraph. Rather than pay Textract's analyze_document+LAYOUT per page, we compose paragraphs ourselves — free, our code, and crucially vendor-neutral because both vendors already normalize into the neutral OcrBlock (coordinates 0..1 of the page). A paragraph's box is the union rectangle of its constituent line boxes; its confidence is their mean; its text is the lines joined; output preserves reading order so list position is a stable ordering. Segmentation is a simple vertical-gap heuristic (a new paragraph starts when the blank space above a line exceeds ~0.6 of that line's height). The pricier native LAYOUT stays a documented future option, only if composed segmentation proves poor on real legal documents.

5. One composition site, so the two paths cannot drift

Composition runs at persistence — persist_native_result and persist_ocr_result both call compose_paragraphs(page line blocks) before the upsert. The Textract adapter is untouched (it still returns line blocks — no behaviour change to the OCR recognition path); the enrichment that the bbox requirement explicitly asked for happens once, downstream, for both paths. A page with no line blocks (the text-only pypdfium2 fallback) composes to [], exactly as before.

6. Off-thread, single worker, quiet

LiteParse's parse is synchronous CPU work, so the worker runs it via asyncio.to_thread — the same off-thread pattern the old pypdfium2 call used, and the reason the port's extract is sync. num_workers=1 matches the single-vCPU worker (avoid OCR-thread oversubscription) and quiet=True suppresses LiteParse's stdout timing chatter that would otherwise become log noise.

Verification (system boundary, not component boundary)

Per the "done = a real request against a real DB, including the empty first-run state" rule, the worker's integration tests run against a real RLS-scoped DB on a freshly seeded firm with zero pre-existing page rows — the real first run:

• Born-digital PDF → indexed via the real LiteParse wheel: a pdfium page row whose engine_version is liteparse/*, with paragraph blocks carrying a 0..1 bbox and exact (1.0) confidence in document_pages — and OCR never called. This is the Story's headline acceptance: a real stored → indexed run with paragraph bbox present, proven against the DB, not a mock.
• Vendor switch proven: the same born-digital upload with DOCUMENT_NATIVE_EXTRACTOR=pdfium still reaches indexed (text-only, no blocks) — the fallback works.
• Scanned input → Textract fallback → indexed with paragraph blocks composed from Textract lines — the bbox requirement on the OCR path too.
• Low-confidence OCR → needs_review; oversize → failed with bytes never fetched; redelivery of a terminal document → no-op — all unchanged from Story-523/524.

Unit tests cover each vendor in isolation (LiteParse blocks + native-only safety: no ~/.tesseract-rs created; pypdfium2 text-only) and the composition function (single line → one paragraph; tight lines → union box; large gap → split; reading order; per-page grouping). Full local gate: ruff clean, 1312 tests pass, make openapi produces no diff (no trust-boundary DTO changed), and the Alembic round-trip is clean on a fresh DB (no migration in this Story). The Lambda-target wheel hash (liteparse-2.1.1-cp312-cp312-manylinux_2_28_x86_64) is present in the hashed lock, so the deploy build installs it with no compile.

One sentence to remember: native extraction now sits behind a port with two swappable vendors — the real LiteParse (default, native bounding boxes, run strictly native-only so its Tesseract never fires) and pypdfium2 (text-only fallback) — and one vendor-neutral composition step gives every page, born-digital or scanned, a paragraph box the UI can jump to.