Story 1856: a document we cannot read becomes honestly unreadable — never a silent dead-end
Why this exists (the catastrophe it prevents). The extraction worker was PDF/image-only: its native engines (LiteParse / pypdfium2) read a PDF text layer, and Textract OCRs scanned PDFs and images. A non-PDF, non-image upload — a Word doc, an Excel sheet, a chat export, an unknown binary — produced an empty native pass, fell through to Textract, and Textract raises on a format it cannot read. A raised error in an SQS consumer is a retryable failure, so the message was redelivered, raised again, and eventually landed in the dead-letter queue — a black hole. The document just sat on the Record with no honest status, and the queue silently filled with poison messages. (RFD-DOC-001 B-7.)
unreadable: stored and malware-clean, but no extractable text in V1. We never hand
Textract a format it will choke on, and the document is marked honestly so counsel can re-upload a
readable copy.Satellite view: route by what the bytes actually are
| Content type | Route | Outcome |
|---|---|---|
| PDF (born-digital) | native text layer (LiteParse/pdfium) | indexed |
| PDF (scanned) / image (PNG·JPEG·TIFF) | OCR (Textract) | indexed / needs_review |
| PDF/image that yields no text at all | OCR attempted, empty | unreadable (was failed) |
| DOCX · XLSX · chat export · unknown binary | neither — Textract is never called | unreadable |
The decisions that matter
1. OCR-eligibility is a hard gate before the Textract call
A small closed set — PDF plus the Textract raster formats (image/jpeg·jpg·png·tiff)
— decides eligibility. A type outside it is routed straight to unreadable; Textract is
never invoked, so it can never raise, so the message can never poison the queue. This is the root-cause
shape: the black hole is closed by construction, not patched per-format.
2. unreadable is a terminal, not a failure
unreadable joins _TERMINAL_STATES, so an at-least-once redelivery is acked as
already-done, never re-processed. It is distinct from failed (a real processing error) and
from a raised exception (which would retry). An unreadable document is not
registrable, so no Mode-2 hand-off is enqueued. The status-event row carries the reason.
3. A blank scan is unreadable, not failed
An OCR-eligible image whose OCR returns no pages (e.g. a blank or illegible scan) is now
unreadable — clean bytes, simply no recoverable text — instead of the old
misleading failed.
4. Observability: the route and outcome ride the existing span
Per pramaan-docs ADR-124 we add no new manual span (the per-record consumer span already wraps
extraction). Instead the consumer span is enriched (T2) with two low-cardinality, closed-vocabulary
attributes — extraction.route (native/ocr/unsupported)
and extraction.outcome (indexed/needs_review/unreadable)
— both added to the manual-span KPI allowlist. One trace_id now answers
“which route did this document take, and why is it unreadable?” No filename, no content.
unreadable state. It does not add text extraction
for Office formats (DOCX/XLSX → searchable text): that needs a new office-extraction adapter/dependency
(none exists in the worker, which is egress-restricted and read-only-rootfs), tracked as a follow-up. Until
then, Office uploads are honestly unreadable rather than a silent dead-end.Verification
| Test | Proves |
|---|---|
test_unsupported_type_routes_to_unreadable_without_ocr | a DOCX is routed to unreadable and Textract is never called (no DLQ throw); the row is accounted, the transition reason recorded |
test_empty_ocr_result_is_unreadable_not_failed | an OCR-eligible image with an empty OCR result is unreadable, not failed |
| existing native/scanned/needs-review/oversize/redelivery suite | the PDF + image happy paths and guards are unchanged |
One sentence to remember: we route each upload by what it actually is, OCR only what
OCR can read, and mark everything else honestly unreadable — so a document we cannot
read is visible on the Record instead of vanishing into the dead-letter queue.