Story 2278 Walkthrough - B2d Collation machine: verified bundle split + logical children + undo

Story 2278: the Collation machine — verified bundle split, logical children, undo

WHY. 40–50% of uploads — opponent filings — arrive as bundles: one PDF carrying a pleading, its Index of Documents, and a stack of annexures. Registered as ONE document, the annexures drown the pleading: one card, one identity, sections that can only gesture at what is really five documents. RFD-DOC-004 D25 (CLOSED 2026-07-11) closes the gap with the same trust pattern as the span and citation gates: the model proposes, code proves. Blake's identification pass (B2a, agents #214) TRANSCRIBES the drafter's own index into a bundle_hypothesis; this Story is the functions half that verifies every claimed cut against actual page content and mints the children — or refuses to.

The machine, end to end

Move	Where	What happens
1 · Pre-filter	`src/shared/context/collation.py::collation_prefilter`	Deterministic, config via `get_settings()` (`COLLATION_MIN_PAGES`, `COLLATION_MIN_CHARS_PER_PAGE`). A small document skips entirely; a text-starved scan skips because there is no index text a verifier could ever confirm against. No model call happens for either.
2 · Identification probe	`build_collation_invoke_payload` / `consume_card_stream`	`agent_invoke.v1` with `mode="card"` — the SAME pass B2a shipped the hypothesis on — over the HEAD pages only (~15 + last 2, D25; `COLLATION_HEAD_PAGES`/`COLLATION_TAIL_PAGES`). The stream fold consumes the `document.card` effect's `collation` payload (`{"compound": bool, "bundle_hypothesis": [rows]}`), newly mirrored in `src/shared/agent_contracts/events.py` so drift still fails loudly. The probe card itself is a head-pages partial and is deliberately NOT ingested — only the hypothesis is consumed. Identity in the payload is the broker's; the hypothesis rows are typed defensively (`parse_bundle_hypothesis`) — a claim, never truth.
3 · Deterministic verifier	`verify_cut_points`	Every claimed start page is confirmed against the parent's actual `document_pages` text inside the ±3 snap window (a D25 constant, not config): an annexure cover MARKER (`ANNEXURE B` / `Exhibit P-3` …) verifies on the claimed page, or on a snapped neighbour when it also matches the index title — a nearby cover for a DIFFERENT annexure can never steal a cut; a title match on a sparse page verifies a marker-less cover; a page that looks like the Index of Documents can never verify anything (it names every title and sits inside early snap windows — the perfect false cover). Cuts must land after page 1 and strictly increase. Anything unconfirmable DEGRADES — it never guesses.
4 · Degrade landing	`_persist_degraded_sections`	An unverifiable index entry becomes an `annexure`-kind SECTION inside the parent (the structural kind Story-2271 seeded; get-or-create via `section_kind.service.get_or_create_annexure_kind` for pre-seed firms), drawn from the drafter's own index line: claimed start, bounded by the next hypothesis boundary, clamped into the document. Provenance pins `prompt_version='collation-degrade-v1'`/`model_id='none'` — code drew it, and that pin doubles as the degrade-only idempotency anchor (a redelivered fan-out appends nothing).
5 · Logical child minting	`_mint_children`	Verified cuts tile the parent exactly once: the pleading body ([1, first_cut−1], riding the parent's filename) plus one child per verified annexure ([cut, next_cut−1]). Each child is a `documents` row with `parent_document_id` (ADR-125) + the new `parent_page_start/end` range pointer (mig 0170) — ZERO raw bytes copied; its object key is a never-written slot in the managed `raw/` grammar. Content = the parent's `document_pages` re-persisted under the child renumbered 1..N (never re-OCRed) + fresh `derived/` artifacts via the extraction worker's own `write_derived_artifacts` (a failed S3 put aborts the whole mint). Born `indexed` with its own `extraction_run_id`; `request_registration` queues its ride on the EXISTING outbox rail, so classification/card/segmentation run per child with no new queue. Privilege flag, declared side, and home class inherit from the parent. Recursion cap = 1: a child (`parent_document_id` set) skips collation before any model call.
6 · Ledger accounting	`_record_child_ledger_items`	Children register under the parent's import as `collation_child` items (the zip-expand precedent), born `accounted` — their pages crossed the scan gate inside the parent, there is no landed→settle leg to wait for. `expected_count` and `accounted_count` grow by N together in one UPDATE: the bundle turned out to be N documents and all N are real, so “N of M accounted” stays truthful and the `accounted ≤ expected` CHECK holds by construction.
7 · Undo (SS-29)	`DocumentService.undo_collation` + `POST /app/matters/{matter_id}/documents/{doc_id}/collation/undo`	Children are logical views, so merge-back is: soft-delete every live child (legal-held children block with a 409), stamp the parent `collation_state='undone'` (durable — the machine never auto-resplits a rejected split), flip the ledger items to `rejected/collation_undone` and give the N back on the NUMERATOR only — completeness stays decidable (`expected == accounted + dropped`). The parent is untouched and whole; every page it ever had is still there. OpenAPI + permission-map snapshots committed; `matters:write` gates the route.

Move

Where

What happens

1 · Pre-filter

src/shared/context/collation.py::collation_prefilter

Deterministic, config via get_settings() (COLLATION_MIN_PAGES, COLLATION_MIN_CHARS_PER_PAGE). A small document skips entirely; a text-starved scan skips because there is no index text a verifier could ever confirm against. No model call happens for either.

2 · Identification probe

build_collation_invoke_payload / consume_card_stream

agent_invoke.v1 with mode="card" — the SAME pass B2a shipped the hypothesis on — over the HEAD pages only (~15 + last 2, D25; COLLATION_HEAD_PAGES/COLLATION_TAIL_PAGES). The stream fold consumes the document.card effect's collation payload ({"compound": bool, "bundle_hypothesis": [rows]}), newly mirrored in src/shared/agent_contracts/events.py so drift still fails loudly. The probe card itself is a head-pages partial and is deliberately NOT ingested — only the hypothesis is consumed. Identity in the payload is the broker's; the hypothesis rows are typed defensively (parse_bundle_hypothesis) — a claim, never truth.

3 · Deterministic verifier

verify_cut_points

Every claimed start page is confirmed against the parent's actual document_pages text inside the ±3 snap window (a D25 constant, not config): an annexure cover MARKER (ANNEXURE B / Exhibit P-3 …) verifies on the claimed page, or on a snapped neighbour when it also matches the index title — a nearby cover for a DIFFERENT annexure can never steal a cut; a title match on a sparse page verifies a marker-less cover; a page that looks like the Index of Documents can never verify anything (it names every title and sits inside early snap windows — the perfect false cover). Cuts must land after page 1 and strictly increase. Anything unconfirmable DEGRADES — it never guesses.

4 · Degrade landing

_persist_degraded_sections

An unverifiable index entry becomes an annexure-kind SECTION inside the parent (the structural kind Story-2271 seeded; get-or-create via section_kind.service.get_or_create_annexure_kind for pre-seed firms), drawn from the drafter's own index line: claimed start, bounded by the next hypothesis boundary, clamped into the document. Provenance pins prompt_version='collation-degrade-v1'/model_id='none' — code drew it, and that pin doubles as the degrade-only idempotency anchor (a redelivered fan-out appends nothing).

5 · Logical child minting

_mint_children

Verified cuts tile the parent exactly once: the pleading body ([1, first_cut−1], riding the parent's filename) plus one child per verified annexure ([cut, next_cut−1]). Each child is a documents row with parent_document_id (ADR-125) + the new parent_page_start/end range pointer (mig 0170) — ZERO raw bytes copied; its object key is a never-written slot in the managed raw/ grammar. Content = the parent's document_pages re-persisted under the child renumbered 1..N (never re-OCRed) + fresh derived/ artifacts via the extraction worker's own write_derived_artifacts (a failed S3 put aborts the whole mint). Born indexed with its own extraction_run_id; request_registration queues its ride on the EXISTING outbox rail, so classification/card/segmentation run per child with no new queue. Privilege flag, declared side, and home class inherit from the parent. Recursion cap = 1: a child (parent_document_id set) skips collation before any model call.

6 · Ledger accounting

_record_child_ledger_items

Children register under the parent's import as collation_child items (the zip-expand precedent), born accounted — their pages crossed the scan gate inside the parent, there is no landed→settle leg to wait for. expected_count and accounted_count grow by N together in one UPDATE: the bundle turned out to be N documents and all N are real, so “N of M accounted” stays truthful and the accounted ≤ expected CHECK holds by construction.

7 · Undo (SS-29)

DocumentService.undo_collation + POST /app/matters/{matter_id}/documents/{doc_id}/collation/undo

Children are logical views, so merge-back is: soft-delete every live child (legal-held children block with a 409), stamp the parent collation_state='undone' (durable — the machine never auto-resplits a rejected split), flip the ledger items to rejected/collation_undone and give the N back on the NUMERATOR only — completeness stays decidable (expected == accounted + dropped). The parent is untouched and whole; every page it ever had is still there. OpenAPI + permission-map snapshots committed; matters:write gates the route.

Placement and failure isolation

Collation is a fan-out sub-step (src/shared/context/fanout.py), LAST — after the enrich kick — so the parent is fully registered, sectioned, and enriched before the probe runs, and the B2b precedent's isolation contract applies verbatim: collate_document_context NEVER raises. Any fault — transport, malformed payload, S3, DB — lands on “no split”, which D25 rule 6 makes the always-available fallback (compound:false + annexure sections IS the promotion path). The whole mint (children + page slices + derived pointers + ledger + registration outbox rows + the parent's split stamp + the audit row) commits in ONE transaction with the guards re-checked inside it, so a crash or a racing redelivery can never leave half a bundle or a double split.

A bad cut can never lose a page. Three layers, each proven at pytest: children are page-range VIEWS, so the parent keeps all 30 fixture pages through split and undo; the child plans tile the parent exactly once (asserted page-by-page); an unverifiable boundary becomes a parent section, not a cut. Undo restores the parent-only view with every page intact and the split recorded rejected.

Schema (mig 0170, expand-only)

Change	Why
`documents.parent_page_start/parent_page_end` (int, NULL)	The logical child's range pointer. CHECKs: paired, 1-based, ordered, child-only (`parent_document_id IS NOT NULL`).
`documents.collation_state` (`'split'\|'undone'`, NULL)	The parent-side split record. `undone` is the durable SS-29 rejection.
`document_import_items.input_kind` CHECK widened with `collation_child`	Ledger vocabulary for move 6. The client-facing DTO REJECTS the value at selection (worker-minted only), mirroring how zip children only come from the expand worker.

Change

Why

documents.parent_page_start/parent_page_end (int, NULL)

The logical child's range pointer. CHECKs: paired, 1-based, ordered, child-only (parent_document_id IS NOT NULL).

documents.collation_state ('split'|'undone', NULL)

The parent-side split record. undone is the durable SS-29 rejection.

document_import_items.input_kind CHECK widened with collation_child

Ledger vocabulary for move 6. The client-facing DTO REJECTS the value at selection (worker-minted only), mirroring how zip children only come from the expand worker.

Downgrade works: collation-child ledger rows are deleted before the CHECK narrows (machine accounting, not business truth), then the columns drop. alembic upgrade/downgrade/upgrade + alembic check ran clean against the isolated test DB.

The golden bundle (SS-28/29/30 at pytest layer)

tests/integration/test_collation_machine.py builds a synthetic opponent Statement of Defence: 30 pages, an Index of Documents naming four annexures, covers at pp. 11/19/28, the second index entry off by 2 (claims 17, stamping shifted the cover to 19), the third entry unverifiable. One narrative test proves: the split into body + 3 annexure children with exact ranges (SS-28); the off-index cut snapping to the verified cover, never the claim (SS-30); the unverifiable entry degrading to an annexure section at its claimed range with zero pages lost; children reusing the parent's pages renumbered (no re-extraction; only derived/ keys ever hit S3); ledger counters 1→5 of 5; the recursion cap and the never-resplit stamp (neither rerun reaches the brain); and the undo restoring the parent whole with the ledger honest again (SS-29). A second test proves the promotion path: nothing verifiable → no children at all, every entry a parent section, idempotent on redelivery. tests/unit/test_context_collation.py (21 tests) pins the pure logic: pre-filter, defensive hypothesis parse, marker/snap/order/index-page verifier rules, exact-tiling plans, clamped degrade ranges, stream folds, and the never-raises contract.

Observed red first. The golden-bundle test failed three times during development before it passed, each failure a real defect it caught: the verifier let the Index page itself verify a cut (title tokens + sparse — fixed with the index-page exclusion), the ledger INSERT omitted id (the table has no server default — fixed), and the first undo implementation decremented the DENOMINATOR too, which would have broken import completeness forever (expected == accounted + dropped — fixed to give back the numerator only).

What this deliberately does NOT do

Not done	Why
Ingest the probe card	It is a head-pages partial, not a whole-document card; only the `collation` payload is consumed. The child's own registration produces its real identity.
Copy raw bytes / re-OCR	D25 rule 4. Children point at the parent's WORM raw via the range pointer; their readable content is the page slice + derived artifacts. Child-level raw download/render UX is app-side, out of scope here.
An approval queue	D25 rule 5: auto-split with undo, never HITL-before-split.
Nested splits	Recursion cap 1, checked before the pre-filter.
B3 consumers	Chunks/graph/insight reshapes stay B3 scope (serial link 6+).

Not done

Why

Ingest the probe card

It is a head-pages partial, not a whole-document card; only the collation payload is consumed. The child's own registration produces its real identity.

Copy raw bytes / re-OCR

D25 rule 4. Children point at the parent's WORM raw via the range pointer; their readable content is the page slice + derived artifacts. Child-level raw download/render UX is app-side, out of scope here.

An approval queue

D25 rule 5: auto-split with undo, never HITL-before-split.

Nested splits

Recursion cap 1, checked before the pre-filter.

B3 consumers

Chunks/graph/insight reshapes stay B3 scope (serial link 6+).