Story 2070: CE-18 — the backfill worker (re-drive, never re-implement)

WHY. The Context Engine turns a registered document into a card, chunks, assertions, and graph edges — the substrate Mira and Vega cite. But that whole chain only fires for documents that flow through registration after the chain existed. A staging/demo matter full of documents uploaded before CE — or documents that reached registration_failed / registration_needs_review when Blake was down — has zero cards, zero chunks, zero graph. The catastrophe CE-18 prevents is a second pipeline: someone re-implementing card-ingest-plus-embed-plus-project as a standalone "backfill script" that drifts from the live path and quietly produces different cards. The demo bar (RFD §14.2, "a fully populated backfilled matter") needs those documents enriched — but through the same code, not a fork. So CE-18 does not process anything itself. It re-drives: it hands each un-enriched document back to the one consumer that already knows how to build all of it.

🛰️ Satellite — the one sentence

The backfill worker selects a matter's un-enriched documents in a bounded, rate-capped batch and, for each, mints a fresh dispatchable document.registration.requested outbox row and moves the document to queued — so the existing registration relay + consumer produce cards + assertions + graph and kick chunk+embed, exactly as for a live upload; a rerun re-drives nothing new because a driven document has already left the eligible set.

✈️ Airplane — re-posting the mail, not re-writing the letter

Picture a mailroom (the registration consumer) that opens each incoming envelope and does a fixed, well-drilled routine: read it, file a summary card, cross-reference it, chunk it for the index. That routine is the only place the office knows how to do that work. Now a stack of old envelopes turns up that never got the routine — some were never posted, some came back "undeliverable." You do not hire a second mailroom to re-do the routine differently. You simply re-post the envelopes, a bounded stack at a time, into the same mailroom's in-tray. CE-18 is the clerk who re-posts. It writes a fresh delivery slip (a new outbox row — because the old slip is already stamped "delivered" and would be ignored) and marks each envelope queued so the mailroom treats it as live mail. The mailroom does the rest, identically. Re-post the same stack tomorrow and the clerk skips everything already back in the flow — no double delivery.

🔬 Microscopic — the three moving parts

1. Eligibility — what a re-drive may touch (and what it must not)

In src/shared/context/backfill.py, one bounded query selects the documents worth re-driving. The WHERE is the whole safety argument:

processing_status = 'indexed'          -- extraction finished; there is text to register
AND availability   = 'live'            -- not archived/deleted
AND extraction_run_id IS NOT NULL      -- the consumer's idempotency anchor exists
AND registration_status IN
    ('not_requested','registration_failed','registration_needs_review')
ORDER BY created_at LIMIT :limit FOR UPDATE SKIP LOCKED

A registered document is terminal under the ADR-128 §2 state machine (no outbound transition exists) — it already carries Blake's card, so it is never re-driven. A queued / registering document has a live request outstanding; re-driving it would double-invoke Blake, so it is left alone. That exclusion is the "skip already-processed / in-flight" rule. FOR UPDATE SKIP LOCKED lets two concurrent invocations split the work instead of fighting over the same rows.

2. The re-drive — a fresh slip, a queued envelope, the shared door

Each selected document advances to queued and gets one registration outbox row, written through the same shared helper the live path uses (insert_document_ingest_event) — no forked SQL:

validate_registration_transition(document.registration_status, "queued")
document.registration_status = "queued"
await insert_document_ingest_event(
    session, ..., event_type="document.registration.requested",
    idempotency_key=backfill_registration_idempotency_key(firm_id, doc.id, run_nonce),
    declared_side=document.declared_side, home=document.home,
)

Why a fresh idempotency key. A registration_failed document still carries its original document_registration:<…> outbox row, and that row is already dispatched. Re-using the deterministic key would ON CONFLICT DO NOTHING onto the spent row, and the relay would never send the re-drive — the document would sit in queued forever with no message. A per-invocation run_nonce makes every re-drive a genuinely new, dispatchable row. The row lock + the → queued transition (which leaves the eligible set) stop a single pass — or a rerun — from minting two rows for one document.

Everything downstream is untouched live code: the ingest relay drains the row to the registrar queue, the consumer's retry edge advances queued → registering, invokes Blake, ingests the card (CE-7), normalizes assertions + projects the graph (CE-9/CE-10), rebuilds chronology (CE-11), and kicks the enrich fan-out that chunks + embeds (CE-3). CE-18 adds none of it.

3. The worker — bounded, rate-capped, resumable, admin-triggered

src/workers/context_backfill/handler.py is a thin Lambda. Its event names the firm, the matters, and a per-matter limit (default 25, hard-capped at 100 in the service — the rate cap). It sets firm-admin RLS scope and runs one bounded batch per matter, each in its own transaction:

{"firm_id": "<uuid>", "matter_ids": ["<uuid>", ...], "limit": 25}

Resumable with no cursor: because a driven document moves to queued (out of the eligible set), re-invoking simply picks up the next un-driven documents. Successive invocations drain the matter; the eligible set shrinking is the cursor. The trigger wiring (a manual invoke or an infra-owned EventBridge rule) is Terraform-owned — this repo owns the function, not the schedule.

The acceptance criteria, mapped

Criterion	Where it's proven
Demo matter populated end-to-end (the right documents re-driven, no others)	`test_backfill_redrives_eligible_documents_only` — against a real DB, the three eligible states move to `queued` with one undispatched backfill registration row each; `registered`, not-indexed, and no-`extraction_run_id` documents are untouched. The downstream card/chunk/assertion/graph production is the live consumer chain, proven by its own tests.
Rerun is idempotent	Same test — a second pass re-drives 0 and mints no new outbox row (driven documents are now `queued`, out of the eligible set).
First-run / empty state (AGENTS "done = system boundary")	`test_backfill_empty_matter_first_run_is_noop` — a freshly provisioned matter with zero documents returns a typed zero, never an error.

What CE-18 deliberately does not do

No parallel pipeline. It produces no card, chunk, assertion, or edge itself. Every artifact comes from the live registration consumer + enrich fan-out. CE-18 is a trigger, not a processor.
No migration. It reuses the existing ops.document_ingest_outbox table and the documents state machine; there is nothing new to store.
No new manual span. Each unit of work already opens the registry-approved functions.document_ingest_outbox.enqueue span inside the shared enqueue helper; the batch loop is orchestration covered by Lambda T1 auto-instrumentation (ADR-124 documented exemption — the registry is closed and holds no backfill span).
No re-drive of terminal documents. A registered document is done by construction; a genuinely pre-CE registered-without-card document is out of reach of re-registration (terminal state) and is surfaced honestly by EngineState.partial, not silently reprocessed.

Recall in one sentence. CE-18 re-posts un-enriched documents into the one live ingest chain — a fresh dispatchable registration request per document, moved to queued, in bounded rate-capped batches — so the demo matter fills through the exact same code as a live upload, and a rerun re-drives only what has not already re-entered the flow.