Story 1976: the domain-event relay — EventBridge adapter, freshness horizon, reconcile sweep
Second pramaan-functions slice of RFD-EVENT-001 (EPIC-1973). Story-1975
left facts sitting durably in ops.domain_event_outbox;
Story-1974 (pramaan-infra) provisioned the bus, the queue, and a relay
Lambda shell holding a placeholder artifact. This Story is the muscle
between them: the relay worker that drains the outbox onto
pramaan-events-<env> every minute, plus the two
relay-owned honesty policies — the 72h freshness horizon (EB-15) and the
reconcile sweep over ops.matter_frame_jobs (EB-12/EB-18).
The pipeline position
create_matter txn: outbox row (Story-1975, merged) <- atomic with the fact
-> THIS STORY: relay Lambda, rate(1 minute) cron (cron = Terraform, flag-gated)
1. freshness horizon: stale rows -> skipped_at/skip_reason, never published
2. FOR UPDATE SKIP LOCKED, seq order, batch 50 (cloned drain pattern)
3. PutEvents in chunks of 10 -> pramaan-events-<env>
4. per-entry outcome -> dispatched_at OR last_dispatch_error, bulk UPDATEs
5. backlog-age metric -> Pramaan/DomainEvents (feeds the 2h infra alarm)
6. reconcile sweep: lease-expired running frame jobs -> failed / requeue-once
-> bus rule matter.created -> SQS -> framer_trigger (Story-1977, next)
What lives where
| File | Owns |
|---|---|
src/shared/domain_events/relay.py | EventBridge PutEvents adapter + the drain. Inside the outbox package because the domain-events gate bans put_events everywhere else (EB-10); the bus resource policy restricting PutEvents to this Lambda's role is the structural kill (EB-19). |
src/shared/domain_events/reconcile.py | The matter_frame_jobs sweep — the named failed-marker truth-writer. |
src/workers/domain_event_relay/handler.py | Thin Lambda entry: staff RLS context, one drain + one sweep per invocation, Bug-1356 warm loop. |
.github/workflows/deploy-domain-event-relay.yml | Artifact-only deploy into the Terraform shell, with the D13 CodeSha256 artifact assert. |
The EventBridge adapter (new transport, cloned discipline)
Selection and error-stamping are byte-for-byte the proven outbox drain
(document_ingest_outbox / projection_outbox):
FOR UPDATE SKIP LOCKED in seq order, batch 50,
ONE bulk UPDATE for successes, ONE unnest error-stamp for
failures (ADR-130 DR-6). The transport is new:
- Chunks of 10 — the PutEvents API maximum.
- 256KB pre-publish reject — entry size (Source + DetailType + Detail, UTF-8 bytes) is checked BEFORE the API call; an oversize entry is stamped failed and never sent, so one fat row can never fail its chunk. With the thin typed envelope this is belt-and-braces.
- FailedEntryCount → rows — the PutEvents response is positional, so each
ErrorCodemaps back to exactly one outbox row; failed rows keepdispatched_at IS NULLand retry next minute.
Delivery is at-least-once by construction (publish fires before the
bookkeeping commits); consumers dedupe by event_id — that is
why the envelope carries two keys.
Freshness horizon (EB-15): the stale-restore kill
An RDS point-in-time restore resurrects old UNDISPATCHED rows with their
UUIDs unchanged — the relay would faithfully publish week-old facts into
consumers whose state no longer matches. So before every drain, rows with
occurred_at older than 72h get durable markers:
skipped_at + skip_reason =
'freshness_horizon_exceeded:72h'. Never published, never faked in
memory, never reusing dispatched_at (the 0144 CHECK
constraints enforce the pairing and the dispatched-xor-skipped rule).
72h — not 24h — because the 2h backlog-age alarm pages an operator long
before the horizon could bite a real outage.
Reconcile sweep (EB-12/EB-18): the failed-marker truth-writer
A dead Lambda cannot mark its own failure. Any
matter_frame_jobs row still running with
claimed_at past the lease (default 900s) is swept:
rebuild_needed → requeue ONCE (back to queued,
flag cleared so a second corpse marks failed); otherwise
failed. One sweep covers both corpse paths — process death
after claim (message acked, DLQ never sees it) and DLQ-landed messages
(the alarm is the operator signal; the sweep writes the Postgres truth).
ops.matter_frame_jobs ships
with Story-1977, which had not merged when this Story landed. The sweep
probes to_regclass('ops.matter_frame_jobs') and no-ops loudly
(matter_frame_jobs.sweep_skipped_table_absent) until that
migration exists — the relay cron never crashes on the missing table, and
the sweep activates automatically the moment 1977's migration runs.
Trace + logging invariants
- The batch span
functions.domain_event_relay.batch(ADR-124 registry extension, CONSUMER kind) LINKS every drained row's storedtraceparent/tracestatevialink_from_trace_context; per-record consumer spans (Story-1977) PARENT from the same stored context — one continuous trace id across the async hop (D4/EB-04). - EB-14 logging policy, test-enforced: INFO/WARNING/ERROR carry
event_id+trace_id(plus error codes on failures) ONLY; the envelope detail JSON rides DEBUG and never reaches production log level. - Backlog metric:
Pramaan/DomainEvents / OutboxOldestUndispatchedAgeSeconds, dimensionEnvironment— the exact contract the pramaan-infra 2h backlog-age alarm reads. Published every deployed drain run; metric failure never sinks the drain.
Deploy: artifact into a Terraform shell, SHA-proven
The shell (role, VPC, env, cron, IAM PutEvents boundary) is
Terraform-owned; the workflow only pushes code + layers and merges the
three deployment-identity env keys the shell's
ignore_changes contract cedes to CI. After
update-function-code, the workflow asserts the LIVE
CodeSha256 equals the base64-SHA256 of the exact zip it
uploaded (inline mirror of pramaan-infra
scripts/assert-lambda-artifact.sh, D13) — a green deploy step
is not artifact proof. The rate(1 minute) cron stays
flag-disabled (domain_events_relay_enabled = false) until the
first SHA-proven deploy; flipping it is a one-line pramaan-infra PR.
Test evidence
| Scenario | Test |
|---|---|
| EB-01 drain: SKIP LOCKED, seq order, dispatched marking | test_relay_selects_pending_rows_skip_locked_in_seq_order, test_relay_marks_dispatched_and_clears_last_error |
| PutEvents 10/chunk | test_put_events_chunks_at_ten |
| 256KB pre-publish reject | test_oversize_entry_rejected_pre_publish_and_stamped |
| FailedEntryCount → rows | test_failed_entry_count_maps_back_to_rows, test_publish_batch_returns_positional_failures |
| EB-15 durable skip | test_freshness_horizon_skips_stale_rows_durably |
| EB-04 batch LINK span | test_relay_batch_span_links_stored_trace_contexts |
| EB-14 log hygiene (ids only at INFO+) | test_info_logs_carry_event_id_and_trace_id_only |
| EB-18 sweep: lease-expired → failed / requeue-once | test_sweep_marks_lease_expired_running_jobs_failed, test_sweep_requeues_once_when_rebuild_needed |
| Table-absence guard | test_sweep_noops_gracefully_when_table_absent |