Story 2219: the stamp said “filed”, the shelf was empty — rescuing the documents the Context Engine never learned

🛰 Satellite — the one sentence

The CE-18 backfill grows a second re-drive class: for a registered document that step 8 never carded and CE-3 never chunked, it performs the step-8 fan-out directly — no Blake re-run, no state transition, just the enrichment kick that was swallowed — and it reports the two classes separately as {"requeued": n, "enrich_kicked": m}. The fan-out itself is no longer copied: it is extracted from the consumer into src/shared/context/fanout.py, and both producers call the same function.

✈️ Airplane — the registry clerk, the runner, and the door to nowhere

Picture the court registry. A document arrives; the registrar (Blake) reads it, decides what it is, and stamps the file REGISTERED. That stamp is final — a registrar never re-stamps a stamped file, and the rules forbid it (there is no arrow out of registered).

Behind the registrar stands a runner, whose only job is to carry the stamped file down the corridor to the index room, where clerks cut it into cards and file them so any future search finds it. The runner's instructions are humane and correct: if the index room is locked, do not tear up the file — just come back and say nothing. A stamped document is still a stamped document.

For several months the index room did not exist. Not locked — unbuilt. The corridor ended in a wall. The runner walked down it, found nothing, came back, said nothing. Every file was stamped REGISTERED. Every shelf in the index room was bare. And because the registrar will not re-stamp, and because the night-shift sweeper (the backfill) was told to collect only unstamped files, nobody was ever going to walk down that corridor again on their behalf.

This Story does two things. It gives the sweeper a second list: stamped files with no cards in the index room. For each, the sweeper does not wake the registrar — the stamp is fine, the reading was done, and a second reading costs a brain run. It simply walks the file down the (now-built) corridor itself. And it moves the runner's instructions out of the registrar's office into a corridor of their own, so that the sweeper and the registrar send files down the same corridor and can never drift into two different ones.

🔬 Microscopic — what changed, part by part

1. The fan-out becomes one shared thing (src/shared/context/fanout.py, new)

Step 8 of the registration consumer used to be four private helpers. They are now one exported function with the seams injected, and the consumer calls it through a thin _fan_out(deps, ctx, result) that keeps the one registration-specific job — extracting Blake's card and checking its identity against the document being registered.

fan_out_document_context(seams, *, firm_id, matter_id, document_id,
                        extraction_run_id, card, log_source, log_extra)

  card is not None ──▶ _ingest_card ──(ok)──▶ _normalize_card ──▶ _project_chronology ──┐
        │                    │                                                          │
        │                    └──(rejected: span/OCR gate)─────────────────────────────┐ │
        │                                                                             ▼ ▼
        └──(card is None: the orphan cohort)────────────────────────────────▶ _kick_enrichment
                                                                                      │
                                                                                      ▼
                                                                     FanOutResult(card_ingested,
                                                                                  enrich_kicked)

Three properties are load-bearing. Failure isolation survives the move: every sub-step runs in its own transaction and swallows its fault into a log line, because the document is already registered when the fan-out runs and a rejected card or a dead queue must never un-register it (RFD-CTX-001 §5.1 invariant M-12b). Card ingest gates the card-derived steps but not the kick: a span-rejected card yields no assertions, yet the document's text still belongs in the moat. And card=None is a first-class input — ingest, normalize, and chronology all hang off a card, so with no card the fan-out is the enrichment kick, which is exactly and only what the env gap swallowed.

The consumer's log-line names are preserved byte-for-byte via log_source (document_registration_consumer.enrich_kicked and friends), so no CloudWatch query or dashboard breaks; the backfill emits the same shapes under context_backfill.*. One new line appears where there used to be pure silence: <source>.enrich_kick_skipped_no_queue. That silence was this bug.

2. The kick's outcome is reported, not just logged (FanOutResult)

The old _kick_enrichment returned None in all three cases: sent, skipped (no queue configured), and failed. A caller could not tell a delivered document from a lost one. It now returns a boolean that rides out in FanOutResult and up into the Lambda's response body. Best-effort is preserved — nothing raises — but “best effort” now has to say out loud whether the effort worked.

3. The orphan cohort, as a predicate (src/shared/context/backfill.py)

The second class selects the documents the first class structurally cannot see. Read it as one sentence: a live, indexed, non-privileged document that Blake already registered, that step 8 never carded, and that CE-3 never chunked.

registration_status = 'registered'      -- invisible to the requeue class, by design
AND availability = 'live'
AND processing_status = 'indexed'
AND privilege_flag = false             -- FN-PRIVILEGE: counsel already ruled. Never indexed.
AND NOT EXISTS (context.document_cards for this document)   -- step 8 never ran
AND NOT EXISTS (context.chunks         for this document)   -- "un-enriched", literally
AND NOT EXISTS (open privilege_suspicion inbox item)        -- I-11: counsel is still ruling
ORDER BY created_at LIMIT :limit

Two of those predicates are safety, not selection, and neither was named in the ticket. privilege_flag = false is the one that would have bitten: the consumer's I-11 hold checks only for an open privilege_suspicion item, but once counsel resolves that item with [Mark privileged] the item closes and the durable answer moves to the column. A backfill that checked only the open item would have cheerfully enriched a document counsel had just declared work product — privilege waiver being the worst legal failure in the system. The open-item check is the other half: while counsel is still deciding, enrichment holds, because you cannot un-read a suspected-privileged document out of a moat query.

The two NOT EXISTS markers do different jobs. No card means step 8 never ran for this document, and it is what keeps the class from re-kicking every carded document on every invocation forever. No chunks is the literal reading of “un-enriched” and, crucially, it is self-clearing: once the CE-3 embed worker's chunks land, the document leaves the eligible set and a rerun skips it. That is the resumable cursor, with no extra state — the same trick the requeue class plays with its → queued transition.

4. Idempotency, without a lock

The requeue class takes FOR UPDATE SKIP LOCKED because it mutates state and mints an outbox row; two invocations must not both drive one document into Blake. The enrich class needs no lock and takes none. It writes nothing to the database. Its only side effect is an SQS send, and the embed worker inserts ON CONFLICT (firm_id, document_id, chunk_hash) DO NOTHING — so a double kick, whether from a concurrent invocation or a rerun before the chunks land, is arithmetically harmless. Holding row locks on registered documents across an SQS round-trip would have been strictly worse.

5. One budget, two classes (backfill_matter)

limit caps the whole invocation, not each class. The requeue class draws first — it is the expensive one, spending a Blake run per document — and the enrich class takes whatever remains. A matter with a large requeue backlog therefore cannot fan out an unbounded number of kicks in the same pass, and because both classes are self-draining, the next invocation resumes exactly where this one stopped.

{"firm_id": "...", "matter_ids": ["..."], "limit": 25}
        │
        ├── class 1  requeue      not_requested / registration_failed / registration_needs_review
        │                         → mint fresh outbox row, → queued, relay → consumer → Blake → step 8
        │
        └── class 2  enrich kick  registered ∧ no card ∧ no chunks ∧ not privileged ∧ no open hold
                                  → fan_out_document_context(card=None) → SQS → CE-3 chunk+embed

{"matters": [{"matter_id": "...", "requeued": 2, "enrich_kicked": 1}],
 "requeued": 2, "enrich_kicked": 1}

What this Story does not do

Test evidence