Story-3300 · the living index · pramaan-functions

Sixteen rows, sixteen ticks.

A lawyer hands PRAMAAN a bundle: one petition with fifteen annexures stapled behind it. The knife (Story-3294) cuts it into sixteen real documents. The fan-out (Story-3295) sends sixteen clerks to read them. Each clerk takes its own time.

Until this Story, what counsel saw during that was: nothing. Sixteen files appear in a list with names derived from the carve — "Annexure A-3.pdf" — and then, minutes later and one at a time, some of them quietly get better names. There was no page that said what is in this bundle, no page that said how far along are you, and no artifact at the end that said here is the index of what you filed.

ADR-180 D19: every bundle gets an index — provisional at carve, final from the cards. A draft appears the moment carve lands; each row upgrades to the proper legal title as its card lands; the final index is rendered as a PDF and filed into the family as a derived document (provenance generated; re-made on revert or re-carve). No model writes it. It doubles as the progress surface.

The whole thing is a projection

The index has no table of its own, no queue, no status store, and no new abstraction. It could not have any of those: it is a statement about facts three other modules already committed, and a second copy of those facts is a second thing to get out of sync.

What the index showsWhere it already livesWho wrote it
Which members, in what order, over what pagesdocuments.documents (family_id, annexure_position, parent_page_start/end)materialize.py, at carve
The proper legal title of eachcontext.document_cards.production_card -> 'name'ingest_production_card, per card drop
Which members have been readagents.agent_step_dropconsume.py, when it applied the drop

src/clerk_worker/bundle/index.py reads those three, assembles rows, hashes them, and — if the hash moved — draws a PDF and files it. That is the entire feature.

Seven things that are easy to get wrong, and were

1 · The generation comes from the members, not from the drop

A drop carries a generation. Building the index at the arriving drop's generation means a straggler from a superseded carve renders an index of members that are no longer live. So the generation is read back off the members' own provenance manifest:

SELECT max((family_manifest->>'generation')::int)
FROM documents.documents
WHERE family_id = :bundle AND family_role = 'annexure'
  AND availability = 'live' AND family_manifest->>'schema' = 'bundle_cut.v1'

Committed carve state, never queue order — and it makes "a stale generation must never become the final artifact" structural rather than a check, because a straggler recomputes the CURRENT generation and finds its members not all read.

2 · The card does not write display_name

The obvious implementation reads documents.display_name and lets apply_record_name upgrade it as each card lands. It would have shipped an index that sat at its provisional titles through a fully-read bundle.

registrar.yaml ships apply_record_name: suggest. A card's name is recorded as a PROPOSAL; nothing on this rail writes the column. The evidence was already in the repo — test_bundle_fanout hand-sets display_name before asserting the D18 block, precisely because the card does not — and it was found by watching the integration test fail with the provisional title still on the page.

So the index reads the card directly, with counsel's approved display_name still outranking it wherever it exists. That the index shows a title counsel has not approved is correct: the index is a read surface over what the machine found, not the document's name of record.

2a · …and reading it directly is wrong without a generation gate

Found in review. context.document_cards holds ONE live card per document and is not generation-scoped; the status table is. The main member's document id IS the bundle's, and the knife keys an annexure's id on its page range — so a re-carve preserves ids, and generation 1's card is still sitting in that table when generation 2's draft is assembled.

Ungated, that draft opens with the previous generation's legal title on a row it prints as pending, and does it again on a row whose own card was refused. A wrong title is the worst failure this surface has available: it is what counsel cites from, and it looks exactly like a right one. So a card title is passed only where the generation-scoped outcome says APPLIED; every other state falls back to what the carve itself knew.

3 · Two final cards race, and neither is last

_settle serialises passes per DOCUMENT. Two annexures' final cards are two different documents. Under READ COMMITTED each pass reads a sibling set in which the other's card has not committed, each concludes "not complete", and the bundle never gets its final index.

The two passes collide on no row, so ON CONFLICT has nothing to arbitrate and FOR UPDATE has nothing to lock. A transaction-scoped advisory lock on the BUNDLE is the honest tool — keyed on the bundle alone, never on (bundle, generation), because a lock key containing the value being compared lets two generations take different locks and miss each other (the Story-3290 finding, one rail over). Lock order is fixed and acyclic: document lock first, bundle lock second, nothing takes a document lock while holding a bundle lock.

4 · A refused card must not hang the bundle forever

"Last card" cannot mean "every card APPLIED": one member whose card fails permanently would mean a bundle that never completes and an index that is never filed at all. Completion is every live member has a SETTLED card row — applied, recorded_only or refused. out_of_order_deferred is deliberately not settled; that drop is waiting, not finished. A refused member's row renders unread and says so, which is R5's honest terminal state on a progress surface.

4a · …and the refusal path returns before the refresh

Found in review, and it is the same bug one level down. A drop whose object carries forbidden custom metadata verifies its BODY and fails its RAIL: _settle claims it, stamps the row refused, and returns at that branch — ahead of the apply, the drain, and the index refresh that sits after them.

So the outcome that turns a member unread, and that on the last member turns the whole index FINAL, was the one outcome the index never saw. Nothing later is guaranteed to fix it: the refusal burns the drop_id, so a redelivery of the same bytes never reaches an applier, and the filed PDF stays a draft with a pending row until a re-carve. The refresh belongs on every terminal path, not only the applying one.

4b · The PUT precedes the commit, so the KEY has to move

Found in review. Every refresh of an existing index wrote its bytes over the key the already-committed document row was handing out — before the card, the status row and the new manifest committed. Two failures at once: a download in flight reads a transaction's uncommitted contents, and if the database work then rolls back, the surviving row's checksum_sha256 and manifest describe bytes that are no longer at that key.

The key is now the digest of the bytes it holds, so every render is a NEW object and the live one keeps its live bytes. What switches is object_key, inside the caller's own transaction — it commits or rolls back with everything else. Superseded renders are left in place: they are rebuildable_derived, and a delete-after-commit has no transaction to sit in, which is the same ordering bug pointed the other way.

Filed into the family, and taken by a revert for free

The index document is uuid5(namespace, "{bundle}:{generation}") — derived, so the mint is idempotent with no "does it exist yet?" read spanning the gap; generation-keyed, so a re-carve is a new statement and the old one is retired by a sweep scoped to rows this module minted.

It is a CHILD of the bundle (parent_document_id) with no page range. That link is what files it into the family — and what makes DocumentService.undo_collation take it with the members it describes, since a revert soft-deletes every live child. There is no revert-aware code in the index module at all. What there IS is an availability check paired with the digest guard: a soft-deleted index whose rows hash identically to the retired one must be re-FILED, not skipped, or the family keeps no index forever.

It is deliberately NOT family_role='annexure', and it is excluded from annexure_register by retention_class = 'rebuildable_derived' (ADR-126's own durability split). A generated index is not a tendered exhibit, and listing it beside real annexures would put a machine artifact into counsel's filing index. Its bytes live under derived/, not raw/, for the same reason: it is regenerated every time a row moves.

Zero model calls, proved two ways

A grep of index.py proves nothing on its own — Story-2169's model call lived three imports away from the Lambda that executed it and every CI stayed green. So the check is the repo's own R11 transitive import walker, run from this module as its entry point, plus the literal grep beside it. Planting an OpenRouterAdapter import reddens both.

What a planted revert found that the tests did not

Deleting "state" from the hashed digest payload changed no digest and reddened no test. It was redundant — the state is a pure function of the row states — so it came out. A test whose docstring claimed the state was what separated a draft from its own final was claiming something true for a different reason, which is the same defect one level up.

A second plant mattered more. Deciding completion from "every member has a card ROW in context.document_cards" — plausible, also committed state — passed the entire integration suite. It fails only where a card was REFUSED: settled in the status table, no card row, so a card-count reader waits forever. Four unit tests caught it. The integration suite alone would have shipped it.

Each of the three review fixes above ships the same way. Reverting the generation gate reddens test_a_card_title_shows_only_where_THIS_generations_card_applied and the re-carve integration proof; deleting the refresh from the rail-refusal branch reddens test_a_terminal_rail_refusal_still_moves_the_index; putting the generation-only object key back reddens test_an_update_never_overwrites_the_key_the_committed_row_exposes. All three were run red before they were run green, and each names only its own subject — no other test in the file moved.

The rendered artifact

Index — Writ Petition bundle
final · generation 1 · 17 of 17 read · page 1 of 1
 0  Main document   Petition under Section 9 of the Arbitration …  pp. 1-11    read
 1  Annexure A      Statement of account, 2024-25                  pp. 12-16   read
 2  Annexure B      Board resolution dated 14.03.2024              pp. 17-19   read
 …
16  Annexure P      Correspondence bundle                          pp. 27-31   unread

Drawn with pypdfium2 — already in the lockfile, already the knife's own library — through a text-object helper extracted out of the fetch-link watermarker so there is one copy of the ctypes wide-string dance rather than two. A4, paginated (the knife's ceiling is 200 members; an index that stopped at the bottom of page one would drop most of a large bundle without erroring). No new dependency, no LaTeX, no headless browser.

Where it sits in the pipeline

… 6  apply the ready drop, in step order
   6' drain whatever it unblocked
   6'' refresh the living index          ← this Story, INSIDE the transaction
   ─── commit ───
   7  fan out this bundle's clerks
   8  one progress frame

Inside the transaction, so the card that upgraded a row and the index that shows it commit together — there is no window where a member has been read and the family's index disagrees. And deliberately NOT gated on the arriving drop's step: a card that arrived before its classify is applied inside the drain, so the pass whose OWN step is classify is the pass that made that card live. The digest guard is what keeps the drops that move nothing cheap.

The ceiling, stated

A render or upload failure is counted (outcome=render_failed) and named, never raised. The index is a derived progress surface and the drop that woke it carries real document truth — burning a member's card apply because a PDF would not draw is the wrong trade, and the next drop of the same bundle re-derives all of it. A database failure is not caught: it has already poisoned the caller's transaction.

This pass has no manual span of its own. The closed ADR-124 registry has no ratified name for it and inventing one fails span-registry-gate.py; it runs inside functions.step_drop.consume, so the work is on the trace already. What it adds is the counter — registered in pramaan-docs first, because the metric-registry gate reads that table from main and a local METRIC_SPECS entry is necessary and not sufficient.