Story-2980: two clocks + a liveness sweeper that checks the bucket before failing

RFD-AGT-014 removed the fused read_timeout that killed healthy runs (Bug-2381). In its place, D7 splits two clocks with opposite sizing, and D17 rules that the liveness sweeper is not a bare time predicate: it must look in the bucket before it stamps anything terminal. This story ships the sweeper Lambda (src/workers/agent_run_sweeper/), the shared clock contract (src/shared/agent_run_delivery.py), and the staff RLS bracket that makes a fleet-wide sweep possible at all (migration 0216).

⏱️⏰ Two clocks, two owners, two red checks

Clock	Job	Sizing	Config owner	Goes red via
⏱️ SLA clock	"slower than expected — look at it". Metric/alert ONLY; changes no run state; invisible to counsel.	Tight, p95-based: `AGENT_RUN_SLA_THRESHOLD_S` (default 600 s).	Sweeper Lambda env (this story).	Sweeper load-time clock assertion + the `agent_run_sla_threshold_breaches` alarm (Chore-2986).
⏰ Liveness clock (`deadline_at`)	"never coming back". Terminal FAILED + honest pipeline status (R5), lawyer-visible.	Generous: `deadline_at = invoked_at + invoke_budget_s + delivery_slack_s`, `delivery_slack_s = max(invoke_budget_s, 300)` — Spike-2928 measurement, not formula.	Broker at mint (Story-2981) from `AGENT_RUN_DELIVERY_SLACK_S` / `AGENT_RUN_WRITE_ALLOWANCE_S`; the sweeper only READS `deadline_at`.	M2a floor guard (`assert_slack_covers_write_allowance`, called at mint AND at sweeper cold start) + the `agent_run_awaiting_artifact_has_deadline` CHECK (M2b, Story-2959).

The mint-guard ceiling is 3,600 s read as REMAINING credential life at mint (Spike-2928 P5: role-chained AssumeRole caps at exactly one hour). The shared helper assert_deliverable_at_mint refuses both an under-provisioned run (budget + slack + write_allowance > remaining) and a caller that passes a nominal session length (> 3,600 s) — the "works at 10:00, fails at 10:07" refusal, made honest.

The sweep sequence

EventBridge (5 min, Chore-2986)
  → assert BOTH clocks' config (M2a load-time guard — bad config fails the canary, never a fleet)
  → ONE staff-scoped indexed scan (ADR-130):
        delivery_status='awaiting_artifact' AND deadline_at < now()     (partial index from 0214)
        + SLA-breach count (metric-only) + immortal-row census (M2b, must be 0)
  → per expired row, in the row's OWN firm scope:
        HeadObject expected_key
        ├─ present → M1 self-heal: extend deadline_at (+grace) + outbox
        │            event agent_run.artifact_observed  → consumer path (Story-2982)
        ├─ absent  → one-way CAS: delivery_status='swept', status='failed',
        │            completed_at, error_class, honest error_message (R5)
        └─ probe error → leave for next sweep (a throttled S3 is NOT an absence)
  → after commit: artifact_undelivered_runs per swept run (D6),
    agent_run_sla_threshold_breaches (D7), sweep census counter (D13 — empty is a counter, not silence)

M1 — look before it fails (the highest-value item)

S3 notifications are best-effort. A lost notification means a correct, complete artifact sits in the bucket while a bare time predicate stamps FAILED — receipts lost, findings discarded, the firm pays twice, and the answer sits there for 90 days (adversary A2).

The sweeper HeadObjects expected_key first. Present → the notification's job is done through the decoupled door: one agent_run.artifact_observed domain event inserted via the R7 outbox in the same transaction as a deadline-grace extension. The results consumer (Story-2982) settles terminal truth — digest/size verification and the outcome discriminator are its contract (D15), never the sweeper's.

Why the outbox and not the results queue. Chore-2929's queue policy admits sqs:SendMessage from the S3 service principal ONLY (scoped by bucket ARN + account), and the Chore-2986 sweeper role carries no SQS grant — deliberately: a brain (or any non-S3 writer) that can announce its own completion can announce a run it never finished. The domain-event outbox is the only decoupled door the sweeper is allowed through, and its idempotency key (<detail-type>:<firm>:<run>) makes a repeat observation a no-op. Coordination note for Story-2982: subscribe to agent_run.artifact_observed (EventBridge, detail = firm_id + subject run_id + trace context) in addition to the SQS results queue.

The RLS gap the sweep could not cross without 0216

agents.agent_run carried FORCE RLS with only firm_isolation_policy: a staff-scoped session reads zero rows, and a fleet sweep would have certified dead fleets as healthy — the exact silent-skip shape D13 exists to catch. Migration 0216 adds the 0144-style staff bracket (staff_sweep_select_policy / staff_sweep_update_policy, SELECT + UPDATE only — the sweeper never inserts or deletes a run). Permissive policies OR together, so firm isolation is unchanged for firm sessions — proven by test, not by reading the policy.

Honest terminal state (R5)

A swept run's user-visible status says WHY: error_class=artifact_delivery_deadline_exceeded and an error_message naming the deadline, the failed verdict, and the retry path — never "Read pending". The stamp is a one-way CAS (WHERE delivery_status='awaiting_artifact'); a row the consumer settled concurrently is left alone, and the 0214 terminal-guard trigger makes delivered/swept absorbing at the database.

Blind-spend counter (D6) and the census (D13)

Seam	Signal
Run swept without ever delivering	`artifact_undelivered_runs` +1 (dimensionless EMF in `Pramaan/AgentRuns` — the alarm contract) + structured log with elapsed time and seat (metric-label cardinality law)
Run past the SLA threshold, still alive	`agent_run_sla_threshold_breaches` count per sweep — alert only, no state change
Sweep finds nothing	`pramaan.agent_run_sweeper.sweep_runs{outcome=empty}` — a counter, not silence
Artifact present at sweep time	`agent_run_sweeper.artifact_present_self_heal` log + outbox event
Probe failure	`agent_run_sweeper.artifact_probe_failed` log; row left for next sweep
Immortal row (awaiting with NULL deadline)	`agent_run_sweeper.immortal_rows_detected` — structurally zero by the 0214 CHECK; counted anyway

Red-first proofs

Planted state	Asserted BEFORE the sweep	Asserted after
Crashed run: no artifact, deadline past	Row IS `awaiting_artifact` (deleting the watchdog fails loudly)	`swept`/`failed`, honest message, completed_at set, terminal guard absorbing, second sweep a no-op
Artifact in bucket + dropped notification	Row IS `awaiting_artifact` with an expired deadline	NOT failed; deadline extended; exactly ONE `agent_run.artifact_observed` outbox row; next sweep empty
Slack that cannot cover the write allowance	—	Refused at mint (`DeliveryClockConfigError`) and at sweeper load time
S3 outage during sweep	—	Probe error counted; the unchecked run is NEVER failed
Two firms with expired runs	—	Staff scan sees the fleet; firm scope still sees only its own (0216 is permissive-OR, not a widening)
awaiting_artifact with NULL deadline	—	INSERT rejected by `agent_run_awaiting_artifact_has_deadline` (M2b unrepresentable)

Deploy shape

.github/workflows/deploy-agent-run-sweeper.yml follows the ADR-162 boundary: pramaan-infra #2986 owns the shell, role, env, VPC, schedule, and alarms; functions CI owns the reviewed artifact, immutable versions, and the live alias. The candidate canary is non-vacuous (R4): it runs the M2a clock assertion, the REAL staff-scoped fleet query, and a HeadObject against the runs bucket (a deliberately-absent key must read as absent; AccessDenied → FunctionError → red deploy). A pre-deploy step asserts the approved OTel floor and the registered OTEL_SERVICE_NAME=pramaan-functions-agent-run-sweeper (ADR-124 D5) on the Terraform shell.