Story 1606: derived/document.md — the markdown the Registrar seam reads
Why this exists (the catastrophe it prevents). Blake — the Registrar agent that classifies a freshly-indexed document (name, type, tags, duplicate?) — takes one field as its raw material: RegistrarDocumentInput.document_markdown. Its prompts assume markdown: heading and paragraph structure is the signal that tells it "this is the title", "this is a clause". But the extraction worker, today, writes exactly two derived artifacts and neither is markdown: text.txt is deliberately plain (it is the literal string fed to to_tsvector, so it must never gain markup or the search vector would diverge), and pages.json is structured JSON for the geometry/highlight UI, not prose. So the seam has a contract hole: the producer never emits the one shape the consumer needs. Without this Story, the Blake broker-consumer (P3b) would have nothing to read, or — worse — we would be tempted to lie and feed plain text as "markdown" (plan Option B), corrupting Blake's structure-dependent classification. This Story closes the hole at the source: synthesize a real document.md from the structure we already captured, at the same terminal transition that writes the other two.
2026-06-22-blake-seam-connection-plan.md §2 Decision: Option A — add the artifact, keep Blake's contract honest, pay one extra S3 put_object. P1 of the seam (parent EPIC-1536, Spike #1594). No new dependency (pure synthesis), no new observability span (ADR-124 auto/caller-span floor — the write rides the worker's terminal-transition unit of work + boto3 T1 auto-instrumentation, exactly like text.txt). The artifact key is the contract; the renderer is swappable (a native LiteParse markdown renderer can later sit behind the same key).🛰️ Satellite: one terminal transition, now three artifacts
extraction worker — text-bearing terminal transition (processor.py)
│
│ pages: Sequence[DocumentPage] (each page: text + paragraph blocks w/ kind+bbox)
│ full_text: str (page text joined — the search-vector source)
│
└─▶ write_derived_artifacts(...) → firms/{f}/matters/{m}/documents/{d}/derived/
├─ pages.json application/json (structured: geometry/highlight)
├─ text.txt text/plain; charset=utf-8 (search-vector source, NOT markdown)
└─ document.md text/markdown; charset=utf-8 (NEW — the Blake/Registrar input) ◀── Story-1606
▲
serialize_document_markdown(pages=pages)
synthesized from the SAME paragraph blocks behind pages.json
The key shape: document.md is built from the same persisted paragraph blocks that pages.json carries — not a second extraction, not a new model. One terminal transition, one deterministic key per artifact, so a re-run overwrites all three in place (idempotent put_object). Three artifacts, one unit of work, one trace.
✈️ Airplane: the change surface (one file of production code)
| File | Role |
|---|---|
src/workers/document_extraction/derived.py | New DOCUMENT_MD_NAME = "document.md" well-known name + _DOCUMENT_MD_CONTENT_TYPE = "text/markdown; charset=utf-8"; new pure serialize_document_markdown(pages); third put_object wired into write_derived_artifacts alongside the existing two. Module docstring documents the new artifact + its rebuildable_derived retention class. |
tests/unit/test_derived_artifact.py | DB-free unit coverage for the renderer: document-order paragraphs, heading-kind → ATX heading, table-kind degrades to text, page-text fallback when a vendor persisted no blocks, empty-block/page skipping, determinism. |
tests/integration/test_document_extraction_worker.py | _assert_derived_artifacts now also asserts document.md lands with text/markdown content-type and a non-empty body — the system-boundary proof for every text-bearing path (native, OCR, needs-review). |
🔬 Microscopic: how the markdown is synthesized
The renderer — title every line by its reason. The blocks were captured at paragraph grain by Story-1584's compose_paragraphs; each block carries a kind (the persisted block_type). The renderer maps each block's kind to a markdown construct, in document order:
def serialize_document_markdown(*, pages):
segments = []
for page in pages: # document order — pages already ordered
blocks = page.blocks or []
if blocks:
segments.extend( # render each block by its kind...
seg for block in blocks if (seg := _markdown_segment(block))
)
else:
text = (page.text or "").strip() # ...or fall back to page text when a
if text: # text-only vendor persisted NO blocks
segments.append(text) # (else the artifact would be empty)
return "\n\n".join(segments).encode("utf-8") # blank line = markdown paragraph sep
_HEADING_BLOCK_TYPES = frozenset(
{"heading", "title", "header", "section_header", "section_heading"}
)
def _markdown_segment(block):
text = (block.get("text") or "").strip()
if not text:
return None # drop empty/whitespace blocks, no blank seg
if block.get("block_type") in _HEADING_BLOCK_TYPES:
return f"# {text}" # heading kinds → ATX heading
return text # paragraph, line, table, ... → plain text
# (this IS "tables degrade to text")
"paragraph" (and upstream "line") on blocks; there is no "heading" kind in the v1 taxonomy (ocr.models.OcrBlock.block_type, paragraphs.PARAGRAPH_BLOCK_TYPE). So for the current corpus document.md is paragraph-flat — which the plan (§7) explicitly accepts: Blake needs name/type/tags/duplicate, not pixel-perfect structure. The heading mapping is written anyway so that the day a layout-aware engine starts labelling headings, the markdown upgrades for free behind the same key — no consumer change.Three properties the synthesis must hold, and why:
| Property | Why it matters |
|---|---|
| Deterministic — same pages → identical bytes | The key is deterministic, so a re-extraction overwrites in place. If the bytes weren't stable, an idempotent re-run would still churn S3 versions. Pure function over ordered input → identical output. |
| Never empty for a text-bearing doc | A geometry-free vendor (text-only) persists no blocks. Synthesizing only from blocks would emit an empty document.md and starve Blake. The page-text fallback is the door that stays open for the zero-blocks first-run state. |
| No new dependency / no new span | Pure stdlib synthesis keeps the Lambda cold-start budget flat; the write is covered by boto3 T1 auto-instrumentation under the existing per-record consumer span (ADR-124 floor), so observability needs nothing new. |
make test-db→ isolated Postgres 16 on:55432, migrated to head.pytest tests/integration/test_document_extraction_worker.py— 6 passed: every text-bearing path (native, OCR/Textract, needs-review) now assertsderived/document.mdpresent,text/markdown, non-empty; the no-extraction path still writes zero artifacts.pytest tests/unit/test_derived_artifact.py— 10 passed (renderer behaviours + determinism).- Full suite: 1364 passed.
ruff checkclean on changed files.mypy src/workers/document_extraction/derived.pyclean. - No route/DTO/trust-boundary change → OpenAPI snapshots unaffected (
make openapi/ codegen not run, per ticket).
One-sentence recall: the extraction worker now writes a third derived artifact, derived/document.md, deterministically synthesized from the same paragraph blocks behind pages.json (heading-kind → ATX heading, everything else → plain-text paragraph, page-text fallback when a vendor has no blocks), so Blake's document_markdown contract is honestly served without a new dependency or span.