Story 1870: tabular extraction — teaching the pipeline to cite “Sheet ‘Damages’, row 247”

Why this exists (the catastrophe it prevents). Opposing counsel produces a 6-sheet Excel workbook: a damages computation, a payment ledger, a parties list. Today a spreadsheet reaches the extraction worker, is not a PDF and not an image, and lands honestly unreadable — safe on the Record, but invisible to search, to evidence-text, and to Blake. The number your client is disputing — row 247 of the “Damages” sheet — cannot be found, quoted, or cited. Worse would be to fake it: run the workbook through a PDF renderer and cite “page 9.” A spreadsheet has no pages; “page 9” is wherever the renderer’s font and paper size happened to break the grid, and a citation that does not match the exhibit counsel is holding is a citation a court can strike. This Story makes spreadsheets and delimited text first-class evidence: one Markdown table per sheet as the single canonical text (no PDF render), and one segment per row carrying its sheet, row, and cell coordinate — so a citation reads “Sheet ‘Damages’, row 247,” exactly what counsel sees.

🛰 Satellite view: a new route, no new schema, no PDF render

S1 (Story-1866) already minted the call number: document_pages grew segment_kind / segment_index / row_index / cell_ref, and owns all RFD-DOC-002 schema in one migration. S2–S8 are logic-only. This Story (S5) is pure logic: a tabular extraction route that sits beside the existing native and OCR routes in the extraction worker.

Zero migrations. Every coordinate this Story writes lands in columns S1 already created. alembic check reports “no new upgrade operations” and the single head f1a2b3c4d5e6 is preserved.

✈️ Airplane view: where the tabular route plugs in

The worker’s readability router was native-first → OCR-fallback → unreadable. Spreadsheets fell off the end into unreadable. The tabular route is checked first, because a spreadsheet is mutually exclusive with a PDF — there is no point paying the (PDF-only) native extractor for bytes it will return empty for:

  content_type
  ------------
  application/pdf ............................. native (born-digital) -> OCR if scanned
  image/png|jpeg|tiff ........................ OCR (Textract)
  spreadsheetml.sheet | ods | csv | tsv ...... TABULAR  (this Story)  <-- checked first
  everything else ............................ unreadable (clean, but not extractable)

The tabular route, like native and OCR, ends in the same finalize sequence — span KPIs, search_vector, the derived/ S3 artifact, the terminal transition, the Mode-2 registration hand-off, the audit row. S5 extracted that tail into one _finalize_text_extraction so all three routes share it; the tabular route is one more caller, not a parallel pipeline.

🔬 Microscopic view: the decisions that matter

1. One segment per row — the row IS the citable unit (DR-8)

A row maps to one document_pages row: segment_kind='sheet', segment_index = the 1-based sheet ordinal, row_index = the 0-based row, cell_ref = the A-column anchor (A1, A247). The S1 resolver turns that into (sheet, 2, row 246, A247), which the renderer reads as “Sheet 2, row 247.” The row text stored is that row as one Markdown table row (| Rent | 1000 |), so it feeds the same per-segment FTS path S6 will index — a search hit lands on the exact row, not a whole-document blob.

The synthetic page_number is a 1-based counter increasing monotonically across all sheets. It satisfies the NOT-NULL/positive page column and gives the upsert its (firm_id, document_id, page_number) conflict key, so a re-run maps each row to the same number and overwrites in place — idempotent, exactly like the OCR/native persisters.

2. Why the DB engine column says pdfium for a CSV

This one looks wrong until you see the constraint. S1 froze the document_pages.engine CHECK to ('pdfium','textract','paddle'), and RFD-DOC-002 doctrine forbids S2–S8 migrations — so a tabular row cannot introduce an openpyxl engine value without a migration this Story is not allowed to write. The column already models the engine family, not the literal library: the port docstring spells out that pdfium is the native-family token and the actual vendor rides engine_version (liteparse/2.1.1 vs pypdfium2/5.10.1). Tabular extends that exactly: engine='pdfium' (native, not OCR), engine_version = the real reader (openpyxl/3.1.5, csv, ods-xml). The honest, high-resolution reader name is always recorded; it just lives in the column built to hold it. The observability span carries extraction.engine='tabular:openpyxl/3.1.5' so traces stay legible too.

3. Every reader is pure-Python and OFFLINE — and the dependency budget is one MIT wheel

The brief allowed openpyxl / csv / MarkItDown’s local path, explicitly NO Azure / LLM / URL / YouTube paths. The lazy-correct read of that:

openpyxl earns the one dependency: hand-parsing XLSX shared-strings/styles is real work, and getting cell types right matters for evidence integrity. It is MIT, and it is not in the pypdfium2/liteparse closure the AGPL/GPL extraction-license guard (Bug-1550) scans — the build stays copyleft-clean.

4. data_only means cached, not recomputed — flagged, not trusted (acceptance criterion)

openpyxl in data_only mode returns the value Excel last cached for a cell; it never recomputes a formula. A workbook authored programmatically and never opened in Excel can have formula cells with no cached value — they read back empty. So the result sets cached_values_only=True for XLSX: the values are reported as-cached, and the caller does not trust them as freshly recomputed. We read what is on disk, honestly labelled, rather than executing spreadsheet formulas in the worker.

5. Caps surface truncation; they never hide it (acceptance criterion)

A pathological sheet (millions of rows) must not exhaust the worker’s bounded memory, but “never silent truncation” is the harder half. A sheet over max_rows/max_cols is read up to the cap, the sheet is marked over_cap, the canonical Markdown gains a visible > ⚠ Truncated… note, and the document routes to needs_review with the reason recorded on the status event. A human is told the table was cut. Truncated-and-flagged, never quietly dropped to zero.

6. A corrupt-but-clean spreadsheet reaches a terminal, not the DLQ (RFD-DOC-001 B-7)

The handler’s contract: a raised exception is a transient failure → SQS retry → eventually the DLQ black-hole. A malformed XLSX (a non-zip declared as a spreadsheet) is the opposite — a permanent, document-specific failure. So the route wraps only the pure parse in a catch and routes a parse failure to the honest unreadable terminal; the DB/S3 work stays outside the catch, so a genuine S3 or Postgres hiccup still raises and retries. A bad file is recorded honestly for re-upload; it never loops.

7. Observability: in-process parsing, covered by the worker’s span (pramaan-docs ADR-124)

Tabular parsing is pure in-process CPU work, exactly like the native/pdfium extractor — covered by the per-record consumer span the handler already opens, so no manual span is added. The only telemetry is T2 enrichment on that existing span via the approved allowlist (extraction.route='tabular', extraction.engine, extraction.outcome) — low-cardinality, never a cell value or filename.

Verification

One-sentence recall: a spreadsheet is a filing cabinet, not a book — S5 files each row into its own drawer/folder/tab coordinate (sheet / row / cell) so a citation reads “Sheet ‘Damages’, row 247,” with one MIT wheel, no PDF render, and zero new schema.