Story 2731: Indigo functions rails - packet producer, broker invoke, envelope persistence

This walkthrough covers the functions half of Indigo (Epic-2624): the deterministic addressable_document_packet.v1 producer, the broker invoke onto the Indigo runtime, and persistence of the returned document_understanding.v1 envelope. Out of scope and unchanged here: the Indigo brain (pramaan-agents #2730), the Terraform runtime deploy (pramaan-infra), and app rendering.

Why this shape

blake-rfd §4.5 and indigo-rfd §9 R1 both settle the same question: the packet is a deterministic slice of the Document Map, built functions-side - "the brain does NOT grow a packet-building pass". So there is no model call anywhere in this change (pramaan-docs ADR-166: lambdas carry, they do not think), and anchor ids are Blake's existing sp1: addresses carried through unchanged. A re-mint would be exactly the rebase blake-rfd forbids, and two "frozen" contracts that disagree on identifiers are not frozen - the first live packet would fail ingest at wiring time, the #2146/#2154/#2177 incident class.

1. Where things live

ConcernFile
Sealed contract mirror + AnchorKind enum homesrc/shared/agent_contracts/indigo.py
Deterministic packet builder + frame prunesrc/shared/context/understanding_packet.py
Invoke, stream fold, persistencesrc/shared/context/understanding_pass.py
document.understanding eventsrc/shared/agent_contracts/events.py
Storage tablealembic/versions/b7e4c1a9f8d3_0194_context_document_understandings.py
Trigger + flagsrc/workers/context_embed/handler.py, src/shared/settings.py

2. The producer validates what the consumer will reject

The consumer (agents/indigo/models.py) is sealed - extra="forbid", frozen, and it byte-matches every anchor against its page text on ingest. This repo mirrors those exact rules and runs them before the invoke, so a defective packet fails here, named, instead of costing a model call and surfacing as an opaque agent-side rejection:

PacketBuildError: packet for document <id> at extraction_version <id> failed the
ingest invariants: anchor sp1:doc-1:1:1:27-82 text does not byte-match page slice

Offsets come from documents.document_pages_v2, whose page text and block offsets are minted together by projection._with_offsets (page-relative, char_end exclusive) - which is why text[char_start:char_end] == block.text holds exactly. The legacy document_pages path is deliberately not a fallback: legacy OCR blocks carry no offsets, so a packet built from them could not satisfy the byte-match invariant. Such a document is skipped (skipped_not_addressable) rather than shipped with anchors we cannot stand behind.

3. AnchorKind: one enum home, and an honest gap

indigo-rfd §9 R1 leaves the AnchorKind set as "residue for the producer child ticket ... frozen in the same functions-side map spec, ONE enum home both repos cite". That home is src/shared/agent_contracts/indigo.py. The Document Map is Textract-layout derived, so the total map is:

Document Map LayoutKindPacket AnchorKind
paragraphblock
tabletable
section_header, titleheading
page_number, figureblock

The map is total: an unknown upstream kind raises rather than defaulting to block, because a new layout kind is a contract event both repos must see, not something to absorb and rediscover months later as degraded reading quality.

Gap found here, closed by Chore-2732 (stacked on this branch). The table above is the LAYOUT axis only. As shipped in this Story, four of the seven kinds - question, answer, exhibit, clause - were not derivable from the Textract-layout Document Map, which would have left every witness finding gate-rejected on testimony documents (§3.5 R2 requires them to anchor to an answer/exhibit span).

Chore-2732 adds a second, TEXT axis in understanding_packet.testimony_anchor_kind: verbatim start-anchored prefixes (^Q\.\s*\d+, ^Ans\., [Witness is (again )?shown ...]) applied only to paragraph blocks on testimony doc_types, proven against all 123 anchors of the Ramanuj gold. clause remains underivable and lands with the contract corpus wave (indigo-rfd D5). What stays true either way: no model, no fuzzy matching - a heuristic Q/A split would mis-attribute testimony, precisely the failure §3.3 R2 exists to prevent.

4. The matter frame carries relevance, never exposure

prune_frame_subset narrows our FrameOutput to the five relevance keys (frame_summary, core_dispute, parties, issues, theory_of_case), recursively dropping nulls, blank strings and empty lists. An entirely empty frame returns None - the block is omitted, never rendered empty (the consumer hard-rejects an empty frame).

The trap this shape hides. Our FrameOutput.parties[].side is us / them / other - that is matter-level alignment, i.e. representing_side wearing a different name, and §8 forbids sending it. FrameParty.role is the party's role in the dispute, mapped from the participant label ("Claimant", "Respondent"); where there is no label we send None. We never derive it from side. The strategy/exposure tier (adversary_best_shot, our_exposure, settlement_posture, value_*, seat, frame_status) is not representable on the sealed model at all, so a regression raises rather than shipping our exposure into an indexable artifact.

5. Invoke rides the existing broker rails

agent_invoke.v1
  agent_key: indigo
  mode:      understand
  meta:      { run_id, firm_id, matter_id, batch_id }   ← tracing only
  input:     { packet }                                  ← the whole input

The runtime ARN is resolved per run from the SSM registry (/pramaan/{env}/agents/indigo/runtime-arn, roster constant AGENT_INDIGO) via the one canonical helper. No AGENT_RUNTIME_ARN_* env var: that N x M env matrix is the #2154/#2177 incident class, and a pinned ARN goes stale the moment the runtime redeploys. Tenant ids stay in meta and out of input - the packet carries no firm_id/matter_id, so nothing outside the sealed object can reach a prompt. Consent, cost-cap and RLS machinery are untouched. Observability reuses the approved AGENT_INVOKE_BUILD/AGENT_INVOKE_CALL spans; no new span name is minted, so no ADR-124 registry amendment is needed.

6. Failure discipline: two classes, two answers

ClassExamplesBehaviour
Indigo answered badly (domain) no product effect, run.failed, contract drift, unstamped provenance, stale extraction_version UnderstandingSeatError; nothing stored, logged loud at context_understanding.read_failed, acked. Re-driven on the next enrich - retrying would re-pay a model that already answered.
Indigo unreachable (infrastructure) AgentRuntimeNotRegistered, AgentRuntimeAccessDenied, transport, stream deadline Propagates - batch-item failure, SQS retries, DLQ eventually shouts. A swallowed capability error is the RCA-006 shape.

Per §3.7 a failed panel or gate emits run.failed and no product effect at all, so status has no error arm. That makes "no effect seen" a failure, and a stream that ends with no terminal event a failure too - silence is never success. partial (rejects > 0 or pages omitted) is a real artifact, not a failure.

7. Storage: idempotent on the triple

Migration 0194 adds context.document_understandings. The idempotency key is deliberately wider than the insight brief's (firm_id, document_id):

UNIQUE (firm_id, document_id, extraction_version_id, prompt_version)

An envelope is only meaningful against the extraction it was read at (every anchor_id addresses that extraction's page text) and under the prompt that produced it. So a re-run of the same triple replaces its row in place, while a re-extraction or a prompt bump writes a new row - the two reads stay comparable instead of one silently destroying the other, which is how the gate reject-rate is measured per-PR.

ColumnNote
understanding JSONBThe envelope stored whole; re-validated on read.
gate JSONBNOT NULL default '{}' - "no stats" and "stats said zero rejects" must not be the same value.
statusCHECK IN ('ok','partial') - no error arm, by contract.
privilegedInherited from documents.privilege_flag at write time; never agent-supplied.
model_id, prompt_versionRequired provenance; an unstamped read is stored by nobody.

RLS is firm isolation + staff-with-grant read, SELECT/INSERT/UPDATE only, DELETE revoked (derived, re-enrichable rows are never deleted).

Caught red before merge. The staff_with_grant_policy helper cloned from migration 0162 omits the RFD-SEC-001 D6 firm-scope clause that migration 0133 applied to every such policy at head. Copying it verbatim reintroduced the pre-0133 unscoped shape on a brand-new table - a grant to firm B could surface firm-B rows inside a firm-A session. test_staff_with_grant_policies_are_firm_scoped_at_head went red (it discovers policies from pg_policies rather than a hand-kept list), and 0194 now carries AND document_understandings.firm_id = app.current_firm_id_or_null().

8. Trigger and flag

The seat runs on the context-enrichment chain beside the Insight seat (context_embed, on the same card/carve-completion eligibility), so a minutes-scale read never rides the registration consumer's budget alongside Blake's passes.

INDIGO_UNDERSTANDING_ENABLED     default false   ← kill switch
INDIGO_UNDERSTANDING_MATTER_IDS  default ""      ← comma-separated allowlist; empty = all matters

Default OFF is required, not cautious: the seat invokes an AgentCore runtime that pramaan-agents #2730 is still building, so an enabled flag against an unregistered runtime would be an AgentRuntimeNotRegistered per document. Off, the seat is fully inert - it never resolves SSM, never invokes, never writes. The allowlist exists because staging should prove a paid per-document model pass on one real matter first. The kill switch wins over the allowlist, so an allowlist is never an accidental enable.

The document.understanding row in pramaan-agents' AgentEventType enum is in flight on #2730. This side is coded against the event-type string and the §3.7 payload shape, and the flag defaults off, so merge order does not matter.

9. Verification

GateResult
ruff check on changed filesclean (121 pre-existing repo-wide errors confirmed identical on main - local ruff 0.15.8 vs CI 0.13.2)
mypyno errors in the four changed source files
pytest tests/unit2414 passed, +31 new; the 20 failures are pre-existing on main (clerk/webhook), verified by stash
make openapino drift (no route/DTO change)
make test-db-alembic-roundtripupgrade → downgrade base → upgrade → alembic check: "No new upgrade operations detected"
Integration (real Postgres, pramaan_app role)4 new store tests + 26 RLS isolation tests pass
R11 model placementtransitive-import gate, placement gate, runtime tripwire, provider-call ban: 19 passed; model-placement-gate.sh passed
Domain events / observability gatesboth passed
Non-vacuous by construction: the builder tests assert that a byte-mismatched anchor, an anchor escaping its page, and an unknown layout kind each raise; the stream tests assert that run.failed, a truncated stream, an unstamped artifact and a smuggled status="error" each fail. The idempotency test would fail under either plausible regression - a missing constraint (duplicate rows) or the narrower (firm_id, document_id) key (count 1 instead of 3).