Story-3290 · step-drop router · pramaan-functions

Delivery order is not semantic order

Blake used to hold a whole bundle in memory and write one artifact at the very end. ADR-180 broke that apart: every step — carve, classify, card, segment — writes its own result to S3 and moves on. That is a strict improvement in durability and a strict downgrade in ordering guarantees, because the path those results now take to the broker is S3 → EventBridge → SQS, which is at-least-once and freely reorderable.

So the consumer this Story ships has to hold one line: the order the drops ARRIVE in tells you nothing about the order they must be APPLIED in. A card that lands before its classify cannot be applied when it lands, and it cannot be thrown away either.

What breaks if you apply on arrival

The card's projection replaces classify's provisional tags and summary — that is the deliberate winner rule, not an accident. Apply them in arrival order and a slow classify overwrites a fast card, so the document ends up holding the provisional answer forever. Search then indexes the wrong thing, and nothing anywhere is red.

The same shape, worse: counsel renames a document between two steps. The human-edit guard lives in the applier, so it holds — but only if the applier is actually the thing that runs. A consumer that "helpfully" writes the card's fields straight onto the row would sail past it.

The four decisions

DecisionWhy not the obvious alternative
Claim each drop by its own drop_id Every transport identifier — the SQS message id, the EventBridge event id, the S3 version id — CHANGES on the redelivery the dedupe exists to catch. Only a key minted by the producing step is stable across it.
A status TABLE, not a log line The table is the refresh path (ADR-180 D22). The living index reads it to render progress and bundle completion is decided from it, generation-aware. A log line can be grepped after the fact; it cannot be joined, read under RLS, or answered inside a transaction.
Defer, then drain Dropping an early card loses work; applying it loses correctness. Recording it as out_of_order_deferred with its own durable object key costs one row and makes the wait visible — and the pass that lands its predecessor picks it straight back up.
Apply through the existing appliers The human-edit guard, V-CITE, the policy gates, the effect ledger and the idempotency claim are all properties of execute_tool_intent and ingest_production_card. Re-implementing the projection would have meant re-implementing all five, and losing one of them silently.

The pipeline

SQS record
  │
  ├─ router.classify_record          → step_drop | clerk_command | unroutable
  │
  ├─ 1. parse_step_drop_key          refuse BEFORE reading the object
  ├─ 2. read_drop_object             GET → recompute digest → compare → parse
  ├─ 3. parse_step_drop              unknown schema_version ⇒ REFUSE, never guess
  ├─ 4. assert_key_matches_body      broker-minted key vs brain-written body
  │
  └─ one transaction:
       5. claim_drop                 INSERT … ON CONFLICT DO NOTHING  (the claim)
       6. is_ready(step, siblings)   classify → card → segment, by APPLIED state
       7a. ready   → apply_drop      → stamp `applied`   → outbox fact
       7b. not yet → stamp `out_of_order_deferred`
       8. drain                      re-drive whatever this drop unblocked
  │
  └─ one run.progress frame          existing v1 vocabulary, additive payload

Every exit names itself. The span's outcome, the status row's outcome and the census log line all carry the same word, and the span opens pessimistically at consume_incomplete. There is no bare return on this path — a drop that did nothing has to say which nothing it did (root AGENTS.md R12).

The race the ordered gate cannot arbitrate

The gate is a read — "are my predecessors applied?" — and a concurrent pass can invalidate the answer before this one commits. Pass A reads the siblings and defers the card because classify has not landed; pass B commits classify and drains, seeing no deferred card because A has not committed its row; A then commits out_of_order_deferred with nothing left to re-drive it. The card is stranded — silently, with every individual step reporting success.

ON CONFLICT cannot fix that: the two passes hold different drop_ids, so they collide on no row and there is no constraint to fire. FOR UPDATE cannot either — the deferred row does not yet exist in the other pass's snapshot. A transaction-scoped pg_advisory_xact_lock on the document is the honest tool: it makes "my read is still true at write time" an invariant instead of a hope, and it releases at commit with no cleanup path to get wrong.

The key is the document, not (document, generation). newest_generation() compares generations, so putting the compared value in the key means N and N+1 take different locks, both proceed, and the older one can commit last over the newer projection. A lock whose key contains the thing being compared arbitrates nothing.

A drop does not get to choose its target

document_id is brain-written and the key↔body check binds only run, step and member. A malformed or compromised drop could therefore name any document in the run's firm and have its card, its segmentation and its proposed effects applied to an unrelated matter's file. The target is now bound to the run row's own matter_id — broker-minted truth — and refused as refused_document_outside_run otherwise.

Stated ceiling, because a check trusted past its edge certifies green: a matter holds many documents, so this narrows the blast radius rather than closing it. Closing it needs the run row to carry the document it was minted for, which belongs to the envelope-minting path (Story-3287).

The carve drop admits it is waiting

Cutting members into real PDFs is the P3 knife (Story-3289 and the EPIC-3286 knife children), not this ticket. A carve drop is therefore recorded and not projected — but it is stamped recorded_only with the reason carve_materialization_pending_story_3289, so "the knife has not landed yet" and "the carve drop vanished" can never look the same in the table.

Two deletions that came with it

The card no longer names the drawer

D21, made real. The card's doc_type is what the paper states, quote-anchored to the document's own words. The drawer label — the firm's catalog-governed filing category — is classify's doc_catalog_label. The broker used to map the first onto the second: it looked the card's phrasing up in the firm catalog and either assigned the match or coined a provisional type from the paper's own wording. That collapses two different questions into one and lets a document name the firm's taxonomy. Where the two disagree, the disagreement is a signal for a human — so the mapping stops rather than picking a winner.

A skipped intent now leaves a row

Before this Story, an applier that declined — human-edit precedence, an unresolvable target — recorded nothing at all. "Blake never proposed a name" and "Blake proposed one and counsel's edit held" produced the same empty ledger. Both now land as a terminal skipped effect carrying the value that did not apply, beside the value that did; and an earlier applied effect that a later step replaces is transitioned to superseded rather than deleted, so the trail answers "what replaced what" instead of only "what is here now". Every row names its source_step and its carve generation.

One bug the tests found

agents.agent_effect carries a BEFORE UPDATE trigger that stamps updated_at = now(), and Postgres now() is transaction start time. The ledger was inserting rows with a Python wall-clock created_at, which is necessarily later. Nothing had ever noticed, because every transition until now happened in a later request — but this rail supersedes an effect recorded moments earlier in the same transaction, and that row's updated_at came out before its own created_at, tripping agent_effect_updated_after_created. Both stamps now come from the database's clock: one row, one clock, and it has to be the clock the trigger uses.

What proves it

ClaimProof
The router fits on one page tests/unit/test_step_drop_router.py counts its executable lines (17 today, tripwire at 60) and asserts it defines exactly one function, holds no AWS client, no session and no await.
Helpers are imported, never copied An AST scan of the whole bundle/ package: the names it DEFINES and the names it IMPORTS from src.shared must not overlap. Plus a named check that ingress.py contains no hashlib — a second digest recomputation would be a forked validator.
Duplicated AND reordered drops converge tests/integration/test_step_drop_pipeline.py runs the same two drops through two firms — forward once each, and reversed with both delivered twice — and asserts the final display name and agent tag set are identical, with exactly two claims on each side.
Counsel's edit survives the later card The card is delivered after a real display_name_user_edited update. The document keeps the human's name, and the refusal is on the record as a skipped effect naming step card and generation 1 — carrying the name that did not land.
The card never reaches the catalog tests/unit/test_card_doc_type_mapping_stopped.py RUNS the projection and asserts assign_document_type / create_matter_document_type are absent — and was watched go RED against a planted reintroduction, because a grep for deleted lines passes just as happily against a reintroduction spelled slightly differently.

Where the seams are