Story 1606: derived/document.md — the markdown the Registrar seam reads

Why this exists (the catastrophe it prevents). Blake — the Registrar agent that classifies a freshly-indexed document (name, type, tags, duplicate?) — takes one field as its raw material: RegistrarDocumentInput.document_markdown. Its prompts assume markdown: heading and paragraph structure is the signal that tells it "this is the title", "this is a clause". But the extraction worker, today, writes exactly two derived artifacts and neither is markdown: text.txt is deliberately plain (it is the literal string fed to to_tsvector, so it must never gain markup or the search vector would diverge), and pages.json is structured JSON for the geometry/highlight UI, not prose. So the seam has a contract hole: the producer never emits the one shape the consumer needs. Without this Story, the Blake broker-consumer (P3b) would have nothing to read, or — worse — we would be tempted to lie and feed plain text as "markdown" (plan Option B), corrupting Blake's structure-dependent classification. This Story closes the hole at the source: synthesize a real document.md from the structure we already captured, at the same terminal transition that writes the other two.

Scope & doctrine. Plan 2026-06-22-blake-seam-connection-plan.md §2 Decision: Option A — add the artifact, keep Blake's contract honest, pay one extra S3 put_object. P1 of the seam (parent EPIC-1536, Spike #1594). No new dependency (pure synthesis), no new observability span (ADR-124 auto/caller-span floor — the write rides the worker's terminal-transition unit of work + boto3 T1 auto-instrumentation, exactly like text.txt). The artifact key is the contract; the renderer is swappable (a native LiteParse markdown renderer can later sit behind the same key).

🛰️ Satellite: one terminal transition, now three artifacts

extraction worker — text-bearing terminal transition (processor.py)
   │
   │  pages: Sequence[DocumentPage]   (each page: text + paragraph blocks w/ kind+bbox)
   │  full_text: str                  (page text joined — the search-vector source)
   │
   └─▶ write_derived_artifacts(...)  →  firms/{f}/matters/{m}/documents/{d}/derived/
          ├─ pages.json     application/json            (structured: geometry/highlight)
          ├─ text.txt       text/plain; charset=utf-8   (search-vector source, NOT markdown)
          └─ document.md    text/markdown; charset=utf-8 (NEW — the Blake/Registrar input)   ◀── Story-1606
                              ▲
                 serialize_document_markdown(pages=pages)
                 synthesized from the SAME paragraph blocks behind pages.json

The key shape: document.md is built from the same persisted paragraph blocks that pages.json carries — not a second extraction, not a new model. One terminal transition, one deterministic key per artifact, so a re-run overwrites all three in place (idempotent put_object). Three artifacts, one unit of work, one trace.

✈️ Airplane: the change surface (one file of production code)

FileRole
src/workers/document_extraction/derived.pyNew DOCUMENT_MD_NAME = "document.md" well-known name + _DOCUMENT_MD_CONTENT_TYPE = "text/markdown; charset=utf-8"; new pure serialize_document_markdown(pages); third put_object wired into write_derived_artifacts alongside the existing two. Module docstring documents the new artifact + its rebuildable_derived retention class.
tests/unit/test_derived_artifact.pyDB-free unit coverage for the renderer: document-order paragraphs, heading-kind → ATX heading, table-kind degrades to text, page-text fallback when a vendor persisted no blocks, empty-block/page skipping, determinism.
tests/integration/test_document_extraction_worker.py_assert_derived_artifacts now also asserts document.md lands with text/markdown content-type and a non-empty body — the system-boundary proof for every text-bearing path (native, OCR, needs-review).

🔬 Microscopic: how the markdown is synthesized

The renderer — title every line by its reason. The blocks were captured at paragraph grain by Story-1584's compose_paragraphs; each block carries a kind (the persisted block_type). The renderer maps each block's kind to a markdown construct, in document order:

def serialize_document_markdown(*, pages):
    segments = []
    for page in pages:                       # document order — pages already ordered
        blocks = page.blocks or []
        if blocks:
            segments.extend(                 # render each block by its kind...
                seg for block in blocks if (seg := _markdown_segment(block))
            )
        else:
            text = (page.text or "").strip() # ...or fall back to page text when a
            if text:                         #    text-only vendor persisted NO blocks
                segments.append(text)        #    (else the artifact would be empty)
    return "\n\n".join(segments).encode("utf-8")   # blank line = markdown paragraph sep
_HEADING_BLOCK_TYPES = frozenset(
    {"heading", "title", "header", "section_header", "section_heading"}
)

def _markdown_segment(block):
    text = (block.get("text") or "").strip()
    if not text:
        return None                          # drop empty/whitespace blocks, no blank seg
    if block.get("block_type") in _HEADING_BLOCK_TYPES:
        return f"# {text}"                   # heading kinds → ATX heading
    return text                              # paragraph, line, table, ... → plain text
                                             #   (this IS "tables degrade to text")
Why heading-kinds today are a no-op — and why that is correct. The paragraph composer only ever stamps "paragraph" (and upstream "line") on blocks; there is no "heading" kind in the v1 taxonomy (ocr.models.OcrBlock.block_type, paragraphs.PARAGRAPH_BLOCK_TYPE). So for the current corpus document.md is paragraph-flat — which the plan (§7) explicitly accepts: Blake needs name/type/tags/duplicate, not pixel-perfect structure. The heading mapping is written anyway so that the day a layout-aware engine starts labelling headings, the markdown upgrades for free behind the same key — no consumer change.

Three properties the synthesis must hold, and why:

PropertyWhy it matters
Deterministic — same pages → identical bytesThe key is deterministic, so a re-extraction overwrites in place. If the bytes weren't stable, an idempotent re-run would still churn S3 versions. Pure function over ordered input → identical output.
Never empty for a text-bearing docA geometry-free vendor (text-only) persists no blocks. Synthesizing only from blocks would emit an empty document.md and starve Blake. The page-text fallback is the door that stays open for the zero-blocks first-run state.
No new dependency / no new spanPure stdlib synthesis keeps the Lambda cold-start budget flat; the write is covered by boto3 T1 auto-instrumentation under the existing per-record consumer span (ADR-124 floor), so observability needs nothing new.
Verify gate (real boundary, not component boundary). Per AGENTS.md "done = system boundary": the proof is a real document driven through the processor against a freshly-migrated Postgres test DB, asserting the artifact lands in S3 — not just a green unit test against a fixture.

One-sentence recall: the extraction worker now writes a third derived artifact, derived/document.md, deterministically synthesized from the same paragraph blocks behind pages.json (heading-kind → ATX heading, everything else → plain-text paragraph, page-text fallback when a vendor has no blocks), so Blake's document_markdown contract is honestly served without a new dependency or span.