Per-page native→OCR, the render-completeness gate & the OCR-page budget

Story-1868 · RFD-DOC-002 S3 (DR-3 / DR-4) · pramaan-functions document extraction + render workers

🛰️ WHY this exists — the catastrophe it prevents. A litigator uploads a 40-page Word brief. Pages 1–38 are typed; pages 39–40 are a photographed exhibit (a scanned signature page). Before this change the pipeline asked one question of the whole document: "does it have a text layer?" Yes → keep it native and the exhibit pages vanish, silently, with no text. Or for a fully-scanned PDF it OCR'd the whole file in one Textract call — but Textract's sync API reads only the first page, so pages 2-N of every multi-page scan were dropped on the floor. Either way the Record shows a clean indexed document that is missing evidence nobody was told about. In a system whose entire promise is "every cited byte is present on the artifact," a silently-empty page is the worst possible failure: it looks like success.

The one analogy: a mailroom that sorts letters, not sacks

The old extractor was a mailroom that judged a whole sack of post by its top envelope: "looks typed, file the sack." This story makes it open the sack and sort per letter — typed letters go straight to filing (cheap, exact native text), photographed letters go to the OCR desk one at a time, and at the end every letter is accounted for. Two guards sit at the doors: a budget (if a sack is nothing but hundreds of photos, hold it for a human rather than running up the OCR bill and timing out) and a completeness check upstream (if the sack was torn open in transit and letters fell out, never stamp it "filed").

✈️ The shape of the change

Two workers cooperate on the processing axis. Crucially, the render-completeness gate (DR-4) already lives in the separate document_render worker (S2) — it runs before extraction and parks an incomplete render at a terminal, so extraction never even sees one. S3's new work is the per-page routing + OCR budget in extraction, plus making the render gate real (S2 wired the seam but never computed the slide count, so the partial gate could never actually fire in production).

  document_render (S2)                         document_extraction (S3, this story)
  ─────────────────────                        ────────────────────────────────────
  stored ─ office? ─ LibreOffice ─ render.pdf      reads the CANONICAL pdf
            │                        │              (rendered.pdf when present,
            │  count_pptx_slides()   │               else the stored object)
            │  → expected_segments   ▼                       │
            │   ┌─ 0 pages/err → unreadable(render_failed)   ▼
            └──▶┤                                  native extract, PER PAGE:
                └─ pages < slides → needs_review     ┌ page ≥ NATIVE_TEXT_MIN_CHARS → keep native (pdfium)
                     (render_partial)                 └ page < floor (scanned)       → OCR that page (textract)
                  ── never indexed on partial ──              │
                                                     OCR pages > budget?
                                                       └ yes → needs_review(ocr_budget)   ← held BEFORE any OCR
                                                       └ no  → carve single-page PDFs, OCR each,
                                                               merge native+OCR rows → indexed

🔬 The three guarantees, each by its reason

DR-3 Per page, so a mixed document loses no page

plan_page_routing() (pure, in routing.py) partitions the native extraction's pages: every page is in exactly one of native_pages / ocr_pages, so "no page silently empty" holds by construction, not by hope. The scanned pages are then OCR'd individually: extract_single_page_pdf() carves page N out of the PDF into standalone single-page bytes (bundled PDFium — no new dependency, no AGPL), Textract reads that one page, and _renumber_page() stamps the result back to the true page number. The persisted page rows then carry their own engine — pdfium for typed pages, textract for scanned ones — because the two upserts key on disjoint page numbers and compose without clobbering.

This also fixes a latent content-loss bug: whole-document sync OCR of a multi-page scan silently kept only page 1. Per-page OCR reads every page.

DR-3 A budget, so a pathological upload is held, not run up

Per-page OCR means one paid Textract round-trip per scanned page. A 2,000-page all-scanned dump would fan out into 2,000 calls and burn the Lambda timeout. So before any OCR runs, if ocr_page_count > DOCUMENT_OCR_PAGE_BUDGET (default 50) the document is held at needs_review(ocr_budget) — whole, with no pages, no derived artifact, no registration. It is never "partially indexed with the rest silently dropped."

DR-4 A completeness gate, so a torn render is never called "filed"

The gate decision (0 pages / error → unreadable(render_failed); rendered pages < expected → needs_review(render_partial)) already lived in the render worker. What was missing — the fixture-green trap — is that the real LibreOfficeRenderEngine never populated expected_segments, so the partial branch could only fire in a test that hand-set it. This story computes the oracle for real: count_pptx_slides() reads a .pptx's OOXML zip and counts its ppt/slides/slideN.xml parts — the same number python-pptx reports, with zero new dependency. A 10-slide deck that renders 3 pages now genuinely surfaces expected_segments=10, and the processor flags render_partial. DOCX has no cheap oracle, so it stays None and only a zero-page render is detectable — exactly as the RFD specifies.

The terminal-reason vocabulary (S1 schema, unchanged)

Reason code	Owning status	Set by	Means
`render_failed`	`unreadable`	render worker	0 pages or render error
`render_partial`	`needs_review`	render worker	rendered pages < slide count
`ocr_budget`	`needs_review`	extraction worker	scanned pages exceed the OCR budget

A DB CHECK couples each code to its owning status, so an unreadable row can never claim a needs_review reason. No migration: S1 already shipped this schema; this story is logic only, single alembic head.

How it was proven — system boundary, not component boundary

DP2-03 — a mixed doc seeded with a rendered_pdf_key + a real 3-page PDF body: typed pages persist pdfium, the scanned page is carved and OCR'd textract, all three pages land, status indexed. (Real DB, real persist + per-segment FTS.)
DP2-04 — three scanned pages, budget 2: held at needs_review(ocr_budget) with zero OCR calls and zero page rows — proving the hold happens before any OCR and is never partial.
DP2-05 / DP2-06 — render worker integration already covers partial/zero; the new engine unit tests prove the real engine now computes the PPTX slide count (10 slides rendered to 3 → expected_segments=10) and that DOCX has no oracle.
Pure unit tests for the routing partition and single-page carve; all existing extraction tests pass unchanged (a born-digital PDF still routes every page native = identical output). No raw text in telemetry — span KPIs stay bounded/non-PII (page count, engine vocabulary, byte size).

✅ One sentence to remember it by. Extraction now sorts the canonical PDF page by page — typed stays native, scanned is OCR'd one page at a time and never lost — held whole if the OCR bill would blow the budget, and a torn render is stopped upstream by a gate that, finally, actually counts the slides.