Founder incident, 2026-07-09: the WebSocket handler refused a Mira
converse send in milliseconds, and the Agent Chat Panel showed
"Mira is thinking… 62s" indefinitely. A refusal rendered as
patience. That inverts the liveness doctrine
(docs/design/ux-guidelines.md §19): the wait must be alive
and honest — a run that stalls must surface, not spin forever
.
The issue assumed a rejection frame arrived and the client ignored it. Reading
the wire (pramaan-functions src/app_api/router_ws.py) shows there
are two different failures wearing one symptom, and only one
of them is a missing frame handler:
| Failure | What the wire actually does | Why the panel hung |
|---|---|---|
| Perimeter rejection (rate limit, input ceiling, circuit breaker, consent block) |
Pushes {"type":"run.failed", payload:{user_visible_message, retryable}} (router_ws.py L852). |
It didn't hang — reduceFrame already handled run.failed. But the error text rode turn.nextMessageMd and landed as an agentText item, so the agent appeared to say "Lost the connection to the agent. Retry." No reference, no retry button. |
websocket_request_rejected(the incident) |
A PramaanError escapes the handler; it returns _error_response(...) to API Gateway (router_ws.py L1004). Nothing is ever pushed to the open connection. |
The client receives no frame at all. Its only defence is the launch watchdog — and the watchdog was unreachable. See below. |
{"type":"heartbeat"} keepalives on a timer
(router_ws.py::post_heartbeat). Every keepalive reset the ceiling.
So a dead run behind a perfectly healthy socket could never trip it — the
panel thought forever. The ceiling existed; it was just unreachable.
A keepalive proves the socket is alive, never the run.
| Acceptance criterion | Shipped |
|---|---|
| Rejected/errored send → immediate error state | New error thread-item kind. A failed run renders a system card (role="alert") — never agent prose. decideDockCommit now returns wait on status === "failed", so error text can no longer masquerade as the agent's reply. |
| Typed-error → human copy map | src/lib/agents/agent-error-copy.ts. Keyed on the real backend error_class values, read from pramaan-functions source: RateLimited, InputTooLarge, CircuitOpen (agent_rails/perimeter.py), InferenceEgressConsentRequired (inference/consent.py), and the coming AgentRuntimeNotRegistered (#2179). An unknown code degrades to reason-copy — a raw class name never reaches counsel. |
trace_id visible + copyable |
The card renders TraceReference (existing primitive, now accepting localized labels). Reference resolution: the frame's trace_id → the run id → the page trace_id meta tag. See the honest gap below. |
| Retry affordance | Rendered only when the backend says retryable. A non-retryable failure (InputTooLarge, AgentRuntimeNotRegistered) offers the reference instead of a button that cannot help. Retry clears error and shows the live thinking state. |
| Client-side thinking ceiling → same state | The watchdog now re-arms only on run progress (isRunProgressFrame), not on heartbeat/pong. Configurable via NEXT_PUBLIC_AGENT_THINKING_CEILING_MS (default 90s, matching the broker's CLERK_STREAM_DEADLINE_SECONDS). Timeout degrades to the same honest card. |
| i18n both locales | 14 keys added to Agents in messages/en.json + messages/hi.json. A unit test asserts every key the copy map can resolve exists in both locales. |
| Storybook states (rejected / timeout / retry-in-flight) | Agents/AgentChatPanel: RejectedRateLimited, RejectedRuntimeNotRegistered, RejectedUntypedFallback, ThinkingCeilingTimeout, RetryInFlight, and an interactive RetryRecovers that replays card → thinking. |
Atlas and Mira are the same connection machinery with different launch
builders. The failure now settles in one place —
failRail() in atlas-rail.ts — which replaced
four hand-rolled copies of the failed-state object in the
store (timeout, replay-exhausted, reconnect-failed, transport-dead). Every
death route produces a RailError, so
agent-dock-mount and mira-dock-mount render the same
card. The incident surface (Mira) and the surface nobody reported (Atlas) were
one bug.
A test that passes both before and after a fix proves nothing. The regression test drives the real store with a fake socket and fake timers, heartbeating every 30s for 5 minutes so the socket never goes 90s silent — the exact incident shape. Reverting the one-line guard makes it fail:
########## PRE-FIX (heartbeat re-arms the watchdog) ##########
× heartbeats do NOT hold the ceiling open — a dead run degrades to an honest error
AssertionError: expected 'running' to be 'failed'
Tests 1 failed | 2 passed (3)
########## POST-FIX ##########
Tests 3 passed (3)
The companion test asserts the opposite direction: real progress frames
(text.delta) arriving just under the ceiling still hold it open,
so a slow AgentCore cold-start completes rather than being killed at 90s.
trace_id is not on this wire yet.
Today's run.failed payload carries only
user_visible_message and retryable. It carries no
error_class and no trace_id. So on a live
rejection right now: the typed-copy map falls back to reason-copy, and the
copyable reference is the run id (the agent-lane handle,
pramaan-docs ADR-146 §7), falling back to the page trace_id meta
tag when the launch was refused before run.accepted. The reducer
reads error_class/error_code/trace_id
forward-compatibly, so the moment functions adds them — the natural home is the
#2179 registry work that introduces AgentRuntimeNotRegistered —
typed copy and a true trace_id light up with no app change.
A functions-side ticket is needed to put those three fields on the
run.failed payload.
router_ws.py (not an invented shape); the
ceiling fix is proven by a test that fails when reverted; and the six panel
states are visually verifiable in Storybook. The remaining unproven step is the
end-to-end staging run. Note the coordination point from the brief: the Mira ARN
fix (#2177) may land mid-build, in which case the natural staging repro becomes a
synthetic bad agent key rather than the original ARN failure.
18 file(s), +886 / -88 lines
| Path | Role | + | - |
|---|---|---|---|
| messages/en.json | Config | +15 | -1 |
| messages/hi.json | Config | +15 | -1 |
| src/components/agents/agent-chat-panel.stories.tsx | Component | +150 | -0 |
| src/components/agents/agent-chat-panel.tsx | Component | +58 | -1 |
| src/components/agents/agent-dock-mount.render.test.tsx | Component | +9 | -1 |
| src/components/agents/agent-dock-mount.tsx | Component | +28 | -1 |
| src/components/agents/agent-thread-item.ts | Component | +28 | -1 |
| src/components/agents/mira-dock-mount.tsx | Component | +27 | -1 |
| src/components/trace-reference.tsx | Component | +16 | -4 |
| src/lib/agents/agent-error-copy.test.ts | Library | +65 | -0 |
| src/lib/agents/agent-error-copy.ts | Library | +53 | -0 |
| src/lib/agents/atlas-rail-store.test.ts | Library | +152 | -0 |
| src/lib/agents/atlas-rail-store.ts | Library | +57 | -58 |
| src/lib/agents/atlas-rail.test.ts | Library | +87 | -1 |
| src/lib/agents/atlas-rail.ts | Library | +111 | -15 |
| src/lib/agents/dock-turn-commit.ts | Library | +10 | -3 |
| src/lib/agents/use-dock-chat-rail.ts | Library | +2 | -0 |
| src/lib/agents/use-mira-conversation-rail.ts | Library | +3 | -0 |
| File | Change |
|---|---|
src/lib/agents/atlas-rail.ts | RailError/RailFailureReason types, RailState.error, failRail(), isRunProgressFrame(); run.failed captures typed code + trace; run.accepted clears a stale error. |
src/lib/agents/atlas-rail-store.ts | Watchdog re-arms only on run progress (the fix); configurable ceiling; four failure objects collapsed onto failRail; launch/resume clear error. |
src/lib/agents/agent-error-copy.ts | New. Typed error_class → i18n key; reason fallback; unknown-code degradation. |
src/lib/agents/dock-turn-commit.ts | A failed run commits nothing — the live error item owns the failure. |
src/components/agents/agent-thread-item.ts | New ErrorItem kind (purely additive). |
src/components/agents/agent-chat-panel.tsx | AgentErrorView — the honest-failure card. |
src/components/agents/{agent,mira}-dock-mount.tsx | Build the error item from rail.error; wire Retry. |
src/components/trace-reference.tsx | Optional localized label/copyLabel/copiedLabel (defaults unchanged for existing call sites). |
src/lib/agents/atlas-rail-store.test.ts | New. Fake-timer ceiling tests, incl. the pre-fix regression repro. |
src/lib/agents/agent-error-copy.test.ts | New. Copy map + both-locale key parity guard. |
messages/{en,hi}.json | 14 Agents keys each. |
add-matter/avatar.tsx, untouched).