Story 2157: CE-18 — the backfill worker gets a door to production

🛰️ Satellite — the one sentence

A new GitHub Actions workflow (deploy-context-backfill-worker.yml) builds the same Lambda artifact the sibling document workers build and creates/updates the pramaan-context-backfill function in the staging VPC — but stripped of every SQS, S3, and Bedrock dependency the registration consumer carries, because the backfill's only side effect is a transactional-outbox write, so it needs nothing beyond DATABASE_URL + VPC + logs/KMS and no event-source mapping: it is operator direct-invoke only.

✈️ Airplane — the same delivery truck, a lighter load

The document workers (extraction, registration consumer) already ride a well-worn delivery route: package src/ + otel_wrapper.py into a zip, attach the shared deps layer and the ADOT OTel layer, drop it into the staging VPC so it can reach RDS, and stamp it with the commit SHA. Story-2157 puts the backfill worker on that same truck — but it doesn't need the same cargo. The registration consumer must be wired to a queue (an event-source mapping), granted S3 read for derived text, and allowed to invoke Blake in AgentCore. The backfill does none of that: it opens a DB transaction per matter, writes outbox rows, and commits. So its role carries only the three permissions a bare in-VPC DB worker needs — write its logs, manage its VPC network interface, and use KMS to decrypt the DB URL — and its trigger is a human pressing "Run workflow" or a direct aws lambda invoke, never a queue.

🔬 Microscopic — the four moving parts

1. The trigger — main-merge on the worker path, plus manual

Unlike the sibling workers (which chain off the Migrate Staging RDS workflow_run), this one deploys whenever the worker code itself changes on main, and on demand:

on:
  workflow_dispatch:
  push:
    branches: [main]
    paths:
      - src/workers/context_backfill/**
      - src/shared/context/backfill.py
      - .github/workflows/deploy-context-backfill-worker.yml

The paths filter keeps unrelated merges from redeploying it, and workflow_dispatch gives the operator the manual button the acceptance criteria call for.

2. The stripped role — three permissions, not eight

The registration consumer's inline policy grants Logs, S3 GetObject, four SQS consume actions, bedrock-agentcore:InvokeAgentRuntime, and KMS. The backfill re-drives purely through the outbox (a DB write the sibling relay then dispatches), so its policy is two statements:

{"Sid":"Logs", ...},
{"Sid":"Kms","Action":["kms:Decrypt","kms:GenerateDataKey"], ...}

plus the managed AWSLambdaVPCAccessExecutionRole for the ENI. No queue ARN to resolve, no bucket, no registrar SSM lookup — those resolution steps are simply deleted from the workflow, not left dangling.

3. The artifact — identical build, validated against the packaged zip

Same package step as its siblings (zip -qr lambda-package.zip src/ otel_wrapper.py), same create-or-update logic, same 1024 MB / 300 s / python3.12 shell in the staging subnets. The validation step imports the handler module out of the extracted zip, so a missing file fails the deploy before it reaches Lambda:

required = {"otel_wrapper.py",
            "src/workers/context_backfill/handler.py",
            "src/shared/context/backfill.py", ...}
importlib.import_module("src.workers.context_backfill.handler")

4. The smoke test — a no-op invoke that proves the wiring

The registration consumer smoke-tests with {"Records":[]}. The backfill's event shape is different, so the smoke payload names a firm but no matters:

--payload '{"firm_id":"00000000-0000-0000-0000-000000000000","matter_ids":[]}'

With an empty matter_ids, the per-matter loop in _backfill_once never runs — no RLS scope is set, no row is touched — so the invoke returns {"statusCode":200,"body":{"matters":[],"redriven":0}}. That is a clean proof that the artifact imports, the cached event loop spins, and the handler is reachable on the live shell, without mutating any firm's data. The real LI-4 evidence invoke (a live firm + matter) is run separately and its outbox rows are recorded on the issue.

5. The IAM propagation race — why the first deploy shipped nothing

The workflow above merged (fn#602) and its first run on main went red, and the Lambda was never created:

aws: [ERROR]: An error occurred (InvalidParameterValueException) when calling the
CreateFunction operation: The role defined for the function cannot be assumed by Lambda.

The role was fine. Its trust policy named lambda.amazonaws.com exactly as written. The problem is that IAM is eventually consistent: the run created the role at 10:45:06 and called CreateFunction at 10:45:14 — eight seconds later — and the Lambda service had not yet seen it. A role that will be assumable in twenty seconds is, at second eight, indistinguishable from a role that never existed.

Think of it as handing someone a business card and immediately asking the front desk to buzz them in: the card is genuine, the desk just hasn't been told yet. The fix is not to forge a better card — it is to knock again.

So .github/scripts/aws-retry-role-propagation.sh wraps the call and retries only that one error string, up to eight times at fifteen seconds:

if ! printf '%s' "$output" | grep -qF "$RETRYABLE"; then
  exit "$status"    # any other aws error fails the deploy immediately
fi

Two properties matter more than the retry itself. First, every other error still fails on the first attempt — a wrapper that retried blindly would turn a real misconfiguration into a two-minute stall ending in the same red build. Second, it propagates the real exit status. The obvious shell spelling of this is a trap:

if output=$(aws ...); then ... ; fi
status=$?   # WRONG: 0, because a false `if` with no `else` exits 0

That bug would have made a failed CreateFunction return success — a green deploy step with no Lambda behind it, which is precisely the "green ≠ live" failure this Story exists to correct. tests/unit/test_aws_retry_role_propagation.py pins all four behaviours.

The race is a class defect, not an instance: all five worker deploy workflows create their role inline and immediately call create-function. The four siblings survive only because their roles were created long ago and their create-branch is now dead code. Per the runtime-liveness doctrine (audit the class, not the reported instance), all five call sites are wrapped in the same change; because the siblings trigger on workflow_run rather than a path filter, patching them ships no surprise redeploys.

What this Story deliberately does NOT do