Story 1500: OcrPort — an engine-neutral OCR seam (Textract impl)
Why this exists (the catastrophe it prevents). OCR is the most engine-specific step in the document pipeline, and we already know we will swap engines: Textract today, PaddleOCR next (privilege/residency — no LlamaParse cloud). If callers learn Textract's response shape — its Blocks array, its 0–100 confidence scale, its geometry — that swap stops being a config change and becomes a rewrite that touches the chronology builder, the review UI, and every overlay. Worse, a single global confidence cutoff would silently mis-gate the new engine the day it ships, because a 0.82 from Textract and a 0.82 from Paddle do not mean the same thing. This story puts a port between the engine and everyone downstream so the engine becomes a swappable detail, and records which engine produced every page so the swap is auditable.
The shape: one port, one neutral result, one persistence home
Three pieces, none of which know each other's internals:
document bytes ──▶ OcrPort.recognize() ──▶ OcrResult (neutral) ──▶ persist_ocr_result() ──▶ documents.document_pages
│ │ (1 row per page)
└ TextractAdapter └ engine + engine_version travel with the result
src/shared/ocr/port.py—OcrPort, an ABC with one methodrecognize(document_bytes, content_type) -> OcrResultand two class attributes (engine,engine_version). The seam every engine plugs into.src/shared/ocr/models.py— the neutralOcrResult/OcrPage/OcrBlock/BoundingBox. Frozen Pydantic models; not HTTP DTOs (they never sit on a route, so OpenAPI hygiene does not apply).src/shared/ocr/textract.py—TextractAdapter, the only place that knows Textract's response shape this version.src/shared/ocr/thresholds.py— the engine-aware confidence gate.documents.document_pages(migration 0110) +persist_ocr_result— the durable home.
What “neutral” means, concretely
| Neutral guarantee | How the Textract adapter satisfies it |
|---|---|
| Geometry is normalized 0–1 of the page — never engine pixel space, so a Paddle box and a Textract box render in the same overlay. | Textract's Geometry.BoundingBox is already 0–1, so it maps straight onto BoundingBox (clamped for tiny float noise). No pixel math. |
| Confidence is 0–1 but engine-relative — compare only within an engine. | Textract reports 0–100; the adapter divides by 100. The value stays engine-relative; the threshold module, not the caller, decides the cutoff. |
| The producing engine travels with the result. | OcrResult.engine + engine_version are stamped from the adapter's class attributes and persisted on every page row. |
| One block per text line, in reading order. | Only Textract LINE blocks become OcrBlocks; WORD/PAGE are dropped. Blocks are grouped by Textract's Page number. |
The engine-specific mapping is split into a pure classmethod TextractAdapter.from_textract_response(response), so the neutrality contract is asserted against fixtures without calling AWS. recognize() is the thin wrapper that calls detect_document_text and hands off to that mapping.
Engine-aware thresholds (the swap-safety knob)
Because confidence is engine-relative, the “is this block reliable?” gate is keyed by the engine that produced the result, read from OcrResult.engine — never hardcoded at the call site.
OCR_CONFIDENCE_THRESHOLDS = { TEXTRACT: 0.80, PADDLE: 0.60 } # paddle ships now, unused until the next adapter
low_confidence_blocks(result) # uses result.engine's gate
A block at 0.70 straddles the two gates: low for Textract (<0.80), fine for Paddle (>0.60). Same number, opposite verdict — that is the whole point, and it is pinned by a test.
Persistence: documents.document_pages (migration 0110)
| Column | Meaning / Mode discipline |
|---|---|
firm_id, document_id | Every row carries firm_id with RLS; FK to documents.documents (ON DELETE CASCADE) and business.firms (RESTRICT). |
engine / engine_version | The producing engine, CHECK IN ('textract','paddle'). This is the audit trail that makes the future Paddle swap traceable per page. |
processing_status | State machine pending → processing → succeeded | failed (CHECK-constrained). No invented doc_state axis — a page's only lifecycle is its OCR state. |
text, blocks | Recognized page text (nullable until success — Mode 1 renders empty/“processing”) and the neutral block array as JSONB (jsonb_typeof = 'array'). |
mean_confidence | Engine-relative mean in 0..1 (CHECK range). Compare only within an engine. |
error | CHECK (processing_status = 'failed') = (error IS NOT NULL) — a failed page must carry a reason; a non-failed page must not. |
Derived data, so upsert — never delete. A page row is fully reconstructable by re-running OCR, so persist_ocr_result upserts on the unique key (firm_id, document_id, page_number): re-OCR overwrites in place instead of accumulating duplicates. The app role is granted SELECT, INSERT, UPDATE and has DELETE revoked, consistent with the firm-owned “never session.delete()” rule. RLS mirrors documents.documents (0031): firm_isolation_policy plus staff_with_grant_policy, and the table is registered in the support-confidentiality list so on-duty support without a content grant sees zero rows.
Observability
No manual span is added. Per pramaan-docs ADR-124 the manual-span registry is a closed enum with no OCR boundary name — adding one would itself need an ADR. The two real boundaries here are covered by the T1 auto-instrumentation floor instead: the Textract call is a botocore AWS-SDK call (auto-spanned), and the page writes are SQLAlchemy statements (auto-spanned). The pure response mapping is covered by the caller's span. The new code is therefore instrumentation-exempt by the auto-instrumentation floor, documented at each module, not by omission.
Acceptance evidence
| Acceptance criterion | Test / check |
|---|---|
OcrPort interface exists | src/shared/ocr/port.py; test_textract_adapter_conforms_to_port |
TextractAdapter conforms (subclass, async recognize, engine attrs set) | test_textract_adapter_conforms_to_port |
| engine + engine_version persisted | test_persist_and_read_back_pages (real DB) asserts both columns on the read-back row |
| thresholds engine-aware | test_thresholds_are_engine_aware, test_low_confidence_uses_result_engine |
contract test on neutral OcrResult shape (0–1 geometry, 0–1 confidence, engine-tagged, multipage) | tests/unit/test_ocr_contract.py |
| Table + migration up/down clean, metadata parity | make test-db-alembic-roundtrip → alembic check: “No new upgrade operations detected” |
System boundary incl. empty first run (zero pages → []) + upsert idempotency | test_persist_and_read_back_pages |
| Support confidentiality registers the new firm table | test_support_app_session_reads_no_confidential_tables |
Review gate — when Paddle ships next version, does anything downstream of the port have to change? No. Callers depend only on the neutral OcrResult; geometry is already 0–1 and confidence is already 0–1, so a PaddleAdapter is a sibling file plus one threshold entry that already exists. The engine column is CHECK-ready for 'paddle', the persisted engine_version keeps the swap auditable per page, and the engine-aware gate prevents the silent mis-gating a global cutoff would cause. Can a page leak across tenants? No — firm_id + RLS + FORCE confine reads/writes; the app role cannot DELETE; ungranted support sees nothing.
Recall: Story 1500 wraps OCR behind an OcrPort so the engine is a swappable detail, returns a single engine-neutral OcrResult (normalized 0–1 geometry, engine-relative 0–1 confidence, engine + version stamped), persists it per page into the firm-isolated documents.document_pages via upsert, and gates confidence per engine — making the Textract→Paddle swap a config change, not a rewrite.