Story-2213 Walkthrough

Change: Build the TTFT/throughput probe that turns RFD-AGT-010's D1 budget (conversational seats need TTFT p50 ≤ 1.5s, p90 ≤ 4s) from a written rule into an enforced, measured gate — then run it against the two seat decisions waiting on it (D2: Sonnet-5 for Mira converse; D3: gpt-5.4-nano for Atlas/Blake). Sibling of the Spike-2055/2081 zdr_probe.py harness, same idiom: stdlib urllib, no deps, raw wire, synthetic fixtures only, never shipped in a CodeZip.

Why a bespoke probe, when OpenRouter publishes latency?

GET /endpoints/zdr returns latency_last_30m per endpoint. Those are fleet-wide aggregates over every caller's prompt shape, effort and prompt length. D1 is a claim about our surface: a ~12k-token Mira-converse sandwich, at the rung's pinned reasoning setting, inside our ZDR provider allowlist. RFD §2 says so outright — "comparative signal only — our eval harness measures OUR TTFT at OUR effort setting before any seat swap" (D6). Hence: our harness, our numbers.

What ships

Not agent code. Lives under docs/notes/, exactly like zdr_probe.py. The brain-egress CI grep polices agents/<agent>/; a probe that makes outbound HTTP must never live there.

A subject is a SEAT, not a model

RFD §4: "Seat = (model, pinned-reasoning-setting, cache_policy, provider-allowlist)". Probing a bare model id would measure something we will never run. So each subject pins all four:

Subject(label="sonnet-low",  model="anthropic/claude-sonnet-5",
        reasoning={"effort": "low"},      # the rung's pinned setting
        only=["amazon-bedrock"],          # the ONE route whose cache probe HIT (0.997)
        sort="throughput")                # §6 constraint order: ZDR → allowlist → sticky → sort

Baselines (deepseek-v3.2, gemini-3.1-flash-lite) are run exactly as built — no reasoning field, no allowlist — because a baseline that silently improves on production is not a baseline.

The measurement's load-bearing decision: cold vs warm

Mira converse is a session. A stable prefix (meta + frame + bundle) is re-sent every turn under a sticky session_id, so turns 2..n read a warm provider cache; turn 1 pays the cold prefix write. That write is a cost counsel pays once per session, not once per turn. Gating a seat on turn-1 TTFT would fail a model for the wrong reason. So:

headline p50/p90 = calls 2..21   (WARM — the steady state counsel lives in)
reported separately = call 1     (COLD — the once-per-session penalty)

This mirrors RFD §3.2's own turn-1-vs-steady-turn cost model.

Two bugs the build caught before they could poison the verdict

1. The --selftest caught a double clock read. now() was called twice for the first content chunk (once for t_first_content, once for t_last_content), timestamping one chunk at two instants and skewing both TTFT and the tok/s denominator. Now: one clock read per delta.
2. The live output caught a stale-cache read. The first sweep printed cold: … cached=16430 — the "cold" call was reading the smoke test's cache from three minutes earlier, because the prefix carried no per-run nonce and stayed byte-identical across processes inside the 5-minute TTL. zdr_probe.py bakes a nonce for precisely this reason. Fixed, verified cross-process, re-confirmed live (cold: … cached=0). Without it, any re-run inside 5 minutes reports warm numbers as cold — the exact "listed ≠ observed" trap this harness exists to close.

The run

export OPENROUTER_API_KEY=<dev key>
python docs/notes/ttft_probe.py --selftest                  # offline asserts, no key, no network
python docs/notes/ttft_probe.py --calls 20 --json out.json  # 5 subjects x 21 calls = 105 streamed calls

Every call carried {"zdr": true, "data_collection": "deny"} and a per-subject sticky session_id. Sequential, never parallel: concurrent calls would contend for one laptop's network and corrupt the very latency being measured.

Observed matrix — LIVE RUN 2026-07-10

Findings

1. D2 — Sonnet-5 fails, and no effort setting can rescue it

The D2 hypothesis was that Sonnet's published 2.8s TTFT is mostly default-medium thinking, so pinning effort:low would pull it under budget. The probe measured 0 reasoning tokens across all 20 warm calls and ttft_content − ttft_any = 0.00s. Pinning worked — the thinking is already gone — and it is still 1.75× over the p50 budget. The 2.63s is pure prefill and network. There is no lower rung to drop to, so this is a permanent verdict, not a "tune it further" one. Sonnet's cache re-confirmed HIT (16,440 tok, warm billed 0.15× cold): cost was never the blocker — TTFT is, exactly as RFD §3.2 predicted.

2. The incumbent straddles the line — and that is the real decision

D2's fallback says "Sonnet fails → flash-lite stays". But flash-lite measured p50 1.54s against a 1.5s budget — over by 0.04s, inside run-to-run noise. A controlled re-run (n=5 warm, fresh prefix) gave p50 1.30s, inside budget. Its p90 of 1.77s is the best of any subject measured.

Read: flash-lite is not failing; it is straddling the line, measured from the wrong machine. Do not swap the converse seat on this evidence. Re-take its p50 from the ap-south-1 runtime first.

3. D3 — gpt-5.4-nano wins the Atlas/Blake everyday rung

Pin effort:none, not low. Both pass, but nano-low emitted 536 reasoning tokens and stretched p90 to 3.74s — 94% of the p90 budget — versus 1.35s at none. The effort menu is rung eligibility (RFD §4.1); low spends budget on thinking the fast rung never asked for. Caveat named, not buried: nano has exactly one ZDR endpoint (Azure).

4. The flash-lite delayed retest — and a counter that lies

zdr-caching-probe-matrix.md left this open: flash-lite MISSed back-to-back, "re-test with a delay before crowning the converse incumbent on an implicit-cache assumption."

First, the counter had to be discredited. In a controlled run with a fully unique bundle body (every paragraph carrying a fresh marker, so no long identical block had ever been sent), Google reported cached_tokens = 10,654 on the very first call — for a prefix that had never existed. It cannot be a prefix-match count. Today's "always cached" and 2026-07-07's "never cached" are the same unreliable field read twice.

So the verdict comes from billed cost — which is what that note itself instructed ("derive it from provider_details['cost'] deltas (billed truth)"):

Verdict: the implicit cache DOES fire on delayed calls (HIT). The original back-to-back MISS was a timing artifact — implicit caches populate after the first response materialises — not a property of the route. The converse seat's cache assumption is sound.

5. A conversational seat without a provider allowlist has no latency class

The as-built deepseek seat pins ZDR and nothing else, so OpenRouter load-balanced it across the 9-provider ZDR set. Sticky routing did not hold: the session drifted SambaNova (8 calls) → DigitalOcean (12 calls) mid-run, and TTFT swung 1.11s – 9.57s. Its cold-vs-warm cost ratio is confounded by the provider swap and must not be read as a cache saving — the two figures were billed by different providers at different rates. This row is the standing argument for RFD §6's constraint order.

Honesty: what this run does not establish

Measured from a dev laptop, not the ap-south-1 AgentCore runtime. Absolute TTFT in production will differ. The laptop's overhead is bounded by the fastest subject on the same machine in the same sweep — nano-none at p50 0.89s — so round-trip cost cannot exceed ~0.9s and cannot explain sonnet's 2.63s. Relative verdicts hold. The one number where this matters is flash-lite's 0.04s p50 margin, which is why finding 2 recommends a re-measure rather than a seat swap.

Verdicts → decisions

Follow-ups (deliberately not done here)

1. Re-measure flash-lite + nano from the ap-south-1 runtime. The converse seat turns on a 0.04s margin taken from a laptop.
2. Give every conversational seat a provider allowlist (§6 constraint 2). Feeds Story-2215 (pinned per-rung reasoning setting in seat config).
3. Never read cached_tokens on a Google route. Cache verdicts derive from billed cost. Worth a guard wherever cache telemetry is consumed.

No seat was swapped in this PR. Story-2213 is a measurement story: it builds the gate and records what the gate says. Acting on D3 (moving Atlas/Blake onto the OpenAI ladder) is Story-2216; pinning the per-rung setting in config is Story-2215.