Story 1607: read_document_markdown — the broker reads the markdown Blake can't

Why this exists (the catastrophe it prevents). Blake — the Registrar agent — classifies a freshly-indexed document from one field: RegistrarDocumentInput.document_markdown. P1 (Story-1606) made the producer honest: the extraction worker now writes derived/document.md. But that markdown lives in S3, and Blake holds no S3 credentials by design (registrar.yaml truth_boundary holds_db_credentials: false; Blake emits intents, the broker executes — it never touches storage). So there is a credential gap: the artifact exists, but the agent that needs it cannot read it. If we closed that gap the wrong way — handing Blake an S3 reader, or letting it presign — we would breach the leash that keeps the brain unable to reach tenant storage directly. This Story closes the gap on the broker side: a read backend the broker-consumer (P3b) calls with its own credentials, firm/matter/document-scoped, to populate document_markdown before invoking Blake. Blake stays credential-free; the broker is the only thing that reads S3.

🛰️ Satellite: the broker reads so Blake doesn't have to

broker-consumer (P3b, later)               Blake / Registrar runtime
   │  RLS session: SET LOCAL                    (NO S3 creds, NO DB creds —
   │  app.current_firm_id = <firm>               emits intents only)
   │                                                      ▲
   └─▶ read_document_markdown(ctx, session,                │ document_markdown
   │       s3_client=<broker creds>, bucket)                │ (populated input)
   │        │                                              │
   │        ├─ _require_document(ctx, session)   ── RLS gate: doc visible to THIS firm?
   │        │       └─ wrong firm → PramaanNotFoundError ("no access")  ── before any S3 call
   │        │
   │        └─ HEAD firms/{f}/matters/{m}/documents/{d}/derived/document.md
   │               ├─ 404 / NoSuchKey ─────────▶ empty contract (present=False)
   │               ├─ size ≤ 32 KiB ── GET ────▶ inline  (markdown string)
   │               └─ size > 32 KiB  ───────────▶ s3 pointer (bucket + key, no GET)
   └──────────────────────────────────────────────────────┘

The key shape: the S3 key is built from the firm/matter/document ids by the same build_derived_prefix the producer uses (Story-1606) — one shared definition, so the reader can never HEAD a key the writer didn't write. The read enforces tenancy twice: the document must be visible under the caller's firm RLS session, and the key is firm-scoped.

✈️ Airplane: the change surface

🔬 Microscopic: HEAD-then-conditional-GET, and the empty contract

Why HEAD first. A legal document's markdown can exceed 32 KiB. We must not load a multi-MB body into the broker's bounded Lambda memory just to decide it should ride S3 instead. So we HEAD for the size, then GET only when it fits inline; a large object returns a pointer to the object that already exists — no re-upload (the producer already wrote it at a deterministic key).

def _read():
    from botocore.exceptions import ClientError
    try:
        head = s3_client.head_object(Bucket=bucket, Key=key)
    except ClientError as exc:
        code = str(exc.response.get("Error", {}).get("Code"))
        if code in {"404", "NoSuchKey", "NotFound"}:
            return DocumentMarkdownRead.empty()   # missing → empty, NOT an error
        raise                                     # AccessDenied/etc. is a real failure
    size_bytes = int(head.get("ContentLength", 0))
    if size_bytes > threshold_bytes:
        return DocumentMarkdownRead(present=True, storage_mode="s3",
                                    s3_bucket=bucket, s3_key=key, ...)   # pointer, no GET
    body = s3_client.get_object(Bucket=bucket, Key=key)["Body"].read()
    return DocumentMarkdownRead(present=True, storage_mode="inline",
                                markdown=body.decode("utf-8"), size_bytes=len(body), ...)

The S3 calls run off the event loop via asyncio.to_thread on the caller's bounded-timeout client (the worker loop contract): a network fault on the VPC S3 endpoint becomes a fast logged error, never an unbounded socket read (the T4 / Bug-1571 lesson).

Two distinct "nothing here" outcomes — and why they must differ.

One-sentence recall: the broker now has read_document_markdown — a credential-bearing, RLS-gated read of derived/document.md that returns the markdown inline under 32 KiB or an S3 pointer above it, treats a missing object as the empty contract and a wrong-firm caller as no-access — so Blake's document_markdown input is populated by the broker while Blake itself never touches S3.