Story 2228: the trapeze with no net — giving the rail deploy something to fall back to

🛰 Satellite — the one sentence

Every rail deploy now publishes an immutable Lambda version before it walks away, and fires the rail canary afterwards to say out loud whether the rail still works — without pretending that saying so protects anyone yet.

✈️ Airplane — the trapeze, the net, and the spotter

Hold one image for the whole walkthrough. A trapeze artist swings between two platforms.

• The artist is the running code that carries a counsel's question to Mira and back.
• $LATEST is the bar the artist is holding right now. There is exactly one bar. update-function-code does not hand over a second bar — it replaces the one in mid-air.
• A published version is a net. It does not stop the fall. It ends it.
• The canary is a spotter standing below, calling out "he's falling."
• A live alias — which the rail does not have yet — is the spotter's hand on the release lever: the thing that lets a shout become a catch.

Before this Story: no net, and no spotter. RCA-006 was a fall onto concrete, watched by a red CI job.

After this Story: a net, and a spotter who shouts. The spotter still has no hand on the lever. That lever is pramaan-docs ADR-161, and it is deliberately not in this diff.

🔬 Microscopic — what actually changed

Three changes to deploy-agent-rail.yml: deploy all three rail Lambdas, publish each version, and record one end-to-end canary verdict.

0. Complete the rail: deploy pramaan-ws-pusher-staging

The existing matrix now includes src.ws_pusher.handler.lambda_handler. Its smoke invokes an empty frame list, proving the packaged runtime imports without sending anything to a browser.

1. The net: Publish immutable version

Slotted into the matrix, after Verify deployed identity and before the clerk smoke test — so the version snapshot captures the code and the merged environment together, already proven to carry the right PRAMAAN_COMMIT_SHA.

VERSION=$(aws lambda publish-version \
  --function-name "$FUNCTION_NAME" \
  --description "$GITHUB_SHA @ $PRAMAAN_DEPLOYED_AT" \
  --region "$AWS_REGION" \
  --query 'Version' --output text)

aws lambda wait published-version-active \
  --function-name "$FUNCTION_NAME" \
  --qualifier "$VERSION" \
  --region "$AWS_REGION"

A published version freezes code and env together and can never be mutated again. Nothing invokes it. That is not a defect — it is the entire point of shipping this slice on its own. The version costs nothing, changes no traffic, and cannot break the rail. It simply means that from this merge onward, every rail deploy leaves behind a thing you can point back at.

2. The spotter: the rail-canary job

A separate job (not a matrix leg — the rail is probed once, end to end, not once per Lambda). It runs scripts/rail-canary-check.sh from Chore-2225, records the exit code, writes the verdict to the job summary, and raises a ::warning:: annotation when the rail is not green.

Two properties of that job are load-bearing, and security review caught both being one careless edit from lost. They are now pinned by tripwires in tests/unit/test_deploy_agent_rail_workflow.py.

The spotter must not shout the key out loud

The fix is an allowlist, not a blocklist — you cannot enumerate the shapes a leaked token might take, so nothing is published unless it is on the list. The script's stdout and stderr go to files that are never printed (stdout is the verdict JSON; stderr is where the script cats a raw FunctionError), and the summary republishes exactly three validated fields.

verdict in                                    summary out
──────────                                    ───────────
result:     "error"                    ──►    result:     error
elapsed_ms: 842                        ──►    elapsed_ms: 842
run_id:     "3977aba1-…"               ──►    run_id:     3977aba1-…
detail:     "connect A failed: …            ✗ withheld
             ?token=eyJhbGciOiJSUzI1…"       (read CloudWatch, where it is access-controlled)

The spotter must not be able to fail the jump

So the job carries continue-on-error: true at the job level — one line, covering every step including the infrastructure steps a per-step flag would miss. That flag comes off in the same change that arms the blocking gate, and not before: the day the canary can genuinely stop a bad deploy is the day it is allowed to fail one.

The honest part: why the spotter has no hand on the lever

The obvious next step — "so gate the deploy on the canary" — does not work, for two reasons that are independent of each other. Both are recorded as TODO(gate) in the workflow itself, where the next engineer will actually read them.

Reason one: the alias governs no traffic

Verified live on 2026-07-10, account 879602786117, ap-south-1:

aws lambda list-aliases returns [] for all three. An unqualified invoke resolves to $LATEST. So by the time the canary job starts, update-function-code has already put the new code in front of counsel. Refusing to shift an alias that nobody consults would turn the job red and leave the bad code running. That is RCA-006 wearing a gate's uniform.

  deploy job                                     canary job
  ──────────                                     ──────────
  update-function-code ──► $LATEST ──► API Gateway ──► counsel
                             ▲                              │
                             │                              │
                       already live                  canary shouts here
                       (before any canary)           ...ten minutes too late

  the alias that would sit between $LATEST and counsel does not exist yet

Reason two: a pre-shift canary certifies the wrong version

This one survives fixing reason one, and it is the subtle half. The canary (src/rail_canary/probe.py) opens a real wss:// socket against the staging WebSocket stage, launches atlas_intake_review, drops the connection mid-run, reconnects, and asserts a terminal frame comes back. It is an end-to-end rail probe. It reaches whatever the stage routes to.

Put an alias in front, and before the shift the stage routes to the old version. A canary run before the alias moves would faithfully certify the code it is meant to be replacing. This is a property of end-to-end probing, not of the wiring — you cannot fix it by rearranging the workflow.

Which is why the gate is an architecture decision, not a YAML edit. pramaan-docs ADR-161 (Proposed) names the only two shapes that exist:

• Option A, shadow rail — a candidate stage, a parallel clerk-runs queue, candidate aliases. The only shape that honours canary → then shift literally. A permanent duplicate subsystem that must stay identical to the real rail or it certifies a lie.
• Option B, shift → canary → auto-rollback (ADR-161's recommendation) — the canary drives the real rail, so it tests exactly what counsel hits. Bad code is exposed for one canary budget (~90s) and the recovery is automatic. Compare RCA-006: ten minutes, and only because a human noticed.

Live proof — green is a build fact, live is a runtime fact

The canary Lambda genuinely does not exist yet, so the new job's real behaviour was exercised against live AWS:

$ GITHUB_OUTPUT=/tmp/gho bash -eo pipefail -c '
    set +e
    ./scripts/rail-canary-check.sh --json-out /tmp/canary.json
    echo "rc=$?" >> "$GITHUB_OUTPUT"'

rail canary is NOT PROVISIONED: no Lambda named 'pramaan-rail-canary-staging' in ap-south-1.
what was needed: a live canary function to prove the rail before this deploy is called done.
what to do next: gh workflow run provision-rail-canary.yml -f environment=staging

--- step exit: 0 ; captured:
rc=3

The script exits 3. The step exits 0. rc=3 is captured, the summary records "UNPROVISIONED", a warning annotation fires, and the deploy is not blocked — which is the whole design, because arming the gate today would block the very deploy that provisions the canary. The verdict branch was exercised for every exit code (0/1/2/3 and an unknown 99); all five return 0, and only the non-zero ones annotate.

What this Story deliberately does not do