A fallback that fires has to say it fired

Nothing pings our seated models. Not once, ever. And the reason that has never hurt yet is the same reason it will hurt badly: pydantic-ai's FallbackModel self-heals on every single call. When Vani's primary dies, her next turn quietly answers on leg 2, then leg 3, and counsel sees a perfectly normal reply. We would be running a different model, at a different price, at a different quality, for days — and the first evidence would be a bill or a complaint, not an alert.

There are two distinct failures hiding in that sentence, and they need two distinct signals, because neither one catches the other.

The rule that shaped the whole design

So the Lambda does exactly what R11 permits a Lambda to do: it triggers the thinking, carries the probe set in, and persists the results out. The probes run on the agent runtime that seats the target.

That is not a workaround. It is the only place the probes can run. The runtime holds the provider keys, the alias→model registry, and — for Atlas, whose seats are ${ATLAS_MODEL_BASE} placeholders by design — the deploy env that expands them. A Lambda could not resolve those three seats even if it were allowed to call a model.

EventBridge rate(1 hour)
   -> seat-health-canary Lambda
        1. derive the probe set from the seat manifest      (49 probes, 21 targets)
        2. one seat_canary.v1 invoke per agent runtime      (6 round-trips)
             ...the runtime runs plain + schema per target, on its own route
        3. classify every result
        4. INSERT one row per probe into ops.seat_health_probe
        5. passive fallback sweep over billing.vendor_cost_event
        6. emit gauges + round census  ->  CloudWatch  ->  #pramaan-alerts

The probe set is derived, and that is the interesting part

A hardcoded probe list would be the exact bug this feature exists to catch: re-crown a seat and the canary keeps certifying the model nobody uses while staying silent about the one they do. So the set is walked out of the seat declarations. Four shapes are live today and every one of them has a test:

panel is declared, never inferred from "is it a list". A panel's three proposers all run in parallel by design. If leg 0 were treated as its primary, the fallback check would report every healthy Vega and Vani build as an incident — every hour, forever, until someone muted it. That is how good alarms die.

The passive half, and the two limits it admits to

The ticket proposed a Logfire query on gen_ai.request.model. The same fact is already durable in our own account: billing.vendor_cost_event.model_id records the model that actually answered, per priced generation. Reading it needs no Logfire token, no egress to logfire.pydantic.dev, and no new IAM or network grant — which under R2 is the difference between a check that ships with its capability and one that is an outage scheduled for later.

The baseline it compares against is the canary's own round. Aliases like deepseek-strong mean nothing to this repo, and Atlas's seats are unresolvable here on purpose — but the runtime already told us what it resolved, and we wrote it down. No second vendored file to drift.

Three places this was built to fail loudly instead of quietly

1. The round is total, or it does not persist

Every probe produces exactly one row, always. A dead runtime becomes an unreachable ROW, not a missing one. A runtime that answers about two of its three seats and drops the third gets an unreachable row for the third. And a round that somehow lost a probe between derivation and persistence refuses to persist at all — a Lambda FunctionError is loud; a short round is silent, and reads as "fewer seats, all fine".

2. unreachable counts toward the alarm

A canary that cannot ping is not good news. If unreachable were excluded from the unhealthy set, the fleet would read green precisely when we are blind to it. It is also its own status, separate from down, because the operator's next move is different: one is a seat swap, the other is an IAM or deploy problem in our own repos.

3. No baseline means no gauge — not a zero

Until a canary round has resolved a model there is nothing to compare traffic against. The fallback check returns measured=False and the Lambda emits no sample, because emitting zero would claim "measured, nothing wrong" about a check that never ran. The alarm is specified with treatMissingData: breaching to match.

Two gauges, and why neither is a counter

Both seat metrics are absolute readings of the current round or window, so both are gauges and both emit zero on a clean pass. An additive instrument fed a live count exports 1, then 2, then 3 for one unchanged broken seat and never returns to zero when it is fixed — the metric lies in the safe direction the moment the incident ends. That has already happened here twice (Bug-3198, Chore-3237) and is recorded as AGENTS.md U12.

The "two consecutive rounds" debounce the ticket asks for is the CloudWatch alarm's evaluation window — two datapoints at a one-hour period. Nothing in this repo has to remember the previous round.

What is red today, and why that is correct

Evidence: every guard was watched failing

Nothing here is claimed green that was not first observed red. The classification and derivation guards were proven by planting reverts and re-running:

The database constraints were proven against real rejected rows on a live Postgres — a healthy row carrying a failure kind, a down row carrying none, an invented status, an invented failure kind, a duplicate probe inside one round, and UPDATE/DELETE as the app role. All seven refused; a legitimate row was accepted.

The seam, stated

seat_chains.yaml is a vendored copy of pramaan-agents' six llm.yaml files. There is no read path between the repos, no published seat manifest, and model_registry.seats() — named in the ticket body — does not exist. Nothing mechanical detects drift; the provenance stamp in the file header is what a reviewer diffs against, and that is a human check.

The repayment is small and named: publish a machine-readable seat manifest to /pramaan/{env}/agents/{name}/seat-manifest on each agent deploy — the rail the runtime ARN already rides. When that lands, this file is deleted in the same PR that reads SSM. Details, with the alarm definitions and the operator runbook, in docs/notes/seat-health-canary-infra.md.