Story 2269: the courier who waited for the hour to strike — immediate outbox dispatch

WHY. A document that takes 15–45 seconds of real work took 2–4 minutes to reach Blake. None of the missing time was compute. It was waiting.

The pipeline hands work across stages through a transactional outbox: the status change and the "please process this" ticket are written in ONE transaction, so a crash can never produce a stored document with no job (or a job with no document). That part is right, and this Story does not touch it. What was wrong is who carried the ticket. Nobody carried it — a relay Lambda woke up on a schedule, every 60 seconds, found whatever tickets had accumulated, and posted them. EventBridge Scheduler's floor is one minute, so the cadence could not simply be turned up.

Two hops pay that toll, so it is paid twice: stored → extraction, and indexed → registration. Worked example from staging (trace 6a5212fe9879e479f0d9e206acd2103c): document reached stored at 09:55:22; extraction began at 09:56:32 — 70 seconds of queue-and-relay wait for about one second of extraction. The lawyer watching the upload spinner has no idea the work finished a minute ago.

The fix in one line

The worker that wrote the outbox row already holds an SQS client. After its transaction commits, it sends its own row — about ten milliseconds later, in the same invocation. The 60-second relay is left exactly as it is, and quietly changes job: from the primary courier to the sweeper that only ever finds rows whose immediate send died mid-flight. 

today:  COMMIT ──(wait ≤60s)──▶ relay ──▶ SQS            p50 stage latency ≈ 30 s
fixed:  COMMIT ──(~10 ms)──▶ same worker sends ──▶ SQS    p50 ≈ 0
             └── relay tick unchanged, sweeps the rare crash case

What actually changed

FileChange
src/shared/document_ingest_outbox.py The existing drain body (_relay_pending_outbox_events) gains an optional document_id narrowing, and a new dispatch_document_outbox_now() runs that same drain for ONE document, in a second transaction opened strictly after the caller's commit. Best-effort: it never raises.
src/workers/document_malware_results/handler.py After the stored transition commits, dispatches the leg it wrote — document.ingest.requested, or document.render.requested for Office docs.
src/workers/document_extraction/handler.py After the indexed transition commits, dispatches document.registration.requested (Blake's hand-off).
.github/scripts/assert-sqs-publish-capability.sh + both deploy workflows The new capability gate, plus the queue-URL env each worker now needs. The malware Lambda also moves off the shared execution role onto the documents pipeline-worker role.
src/workers/document_ingest_relay/handler.py Unchanged. That is the point.

Why a double send cannot hurt

The dangerous-sounding case is: the worker sends the message, then dies before stamping dispatched_at. The relay later finds the row still marked pending and sends it again. Two messages, one document — a duplicate extraction?

No. The outbox row's idempotency_key is passed to SQS as the FIFO MessageDeduplicationId (DocumentIngestQueueClient.enqueue_batch), and that key is deterministic in (event_type, firm_id, document_id). SQS drops the second copy inside its deduplication window. The safety was already in the substrate; immediate dispatch just relies on it.

What is deliberately NOT done. The send is not moved inside the transaction. That would be the classic dual-write bug — a message on the queue for a transaction that then rolls back, pointing at a document that never reached stored. The INSERT stays where it was; only the delivery moved earlier.

The failure path is the interesting path

A failed immediate send is logged and swallowed. The invocation still returns success, because its real work — the status transition — is already committed and durable. The row keeps dispatched_at IS NULL, its last_dispatch_error is stamped, and the unchanged 60-second relay sweeps it on the next tick. The worst case after this change is precisely the behaviour before it.

That is also the trap. A missing IAM grant would be invisible: every send fails silently, the sweeper covers, the pipeline still works — and the latency win simply never lands, with nothing red anywhere. So the grant gets a check that goes red at DEPLOY time, not at runtime: 

$ .github/scripts/assert-sqs-publish-capability.sh \
    pramaan-documents-pipeline-worker-staging \
    pramaan-documents-ingestion-staging pramaan-documents-render-staging

arn:aws:sqs:ap-south-1:...:pramaan-documents-ingestion-staging   implicitDeny
arn:aws:sqs:ap-south-1:...:pramaan-documents-render-staging      implicitDeny
FAIL: pramaan-documents-pipeline-worker-staging cannot sqs:SendMessage to every queue
      it publishes to.                                                    # exit 1

That is the observed-red run against staging as it exists today. The grant itself is a pramaan-infra Terraform resource (pramaan-infra#130), which must apply before this deploys; the gate script ships a planted-negative unit test so the check itself is proven to fail.

Least privilege, not convenience. The malware-results Lambda ran as the shared pramaan-lambda-execution-role — the role app-api, admin-api, and db-bridge also use. Granting that role SendMessage would have handed the public API the power to inject ingestion messages. Instead the Lambda moves onto the documents pipeline-worker role, which is where Terraform already grants it its scan-gate permissions — closing a live-vs-Terraform drift on the way past.

Observability

No new span, no raw tracer call. The immediate drain opens the SAME registry-approved batch span the relay opens (DOCUMENT_INGEST_RELAY_BATCH and siblings), links from the trace context stored on the outbox row, and the send itself is the same boto3 SQS call that sits on the ADOT auto-instrumentation floor (pramaan-docs ADR-124 T1). Instrumentation-exempt by that tiering.

How it is proven

PropertyTest
The committed row is sent immediately and stamped dispatched_at test_immediate_dispatch_sends_the_committed_row_and_stamps_dispatched_at
No double-processing: the relay finds nothing left to resend test_immediate_dispatch_leaves_the_relay_nothing_to_resend
A forced SQS failure is swallowed and the sweeper still delivers test_immediate_dispatch_failure_is_swallowed_and_left_for_the_sweeper
The send is strictly AFTER commit, in both producers test_malware_worker_dispatches_the_stored_legs_after_commit, test_extraction_worker_dispatches_the_registration_leg_after_commit
The capability gate is red without the grant test_capability_gate_is_red_when_the_grant_is_absent (planted negative)

Staging proof (post-merge)

The acceptance bar is wall-clock, so it can only be read after the deploy: upload one document and confirm stored → extraction start < 5 s and indexed → registration start < 5 s, from the status-event timestamps and the trace. Until pramaan-infra#130 applies, the deploy gate holds this change out of staging entirely — which is the intended order.