Change: T2 (#2047) proved Mira's injection wall holds against a
credulous stub — deterministic, offline, no key. This T3 lane is the twin proof against
the real Mira brain: it fires the same corpus at the live converse model and the
live command panel, and a judge model grades the fuzzy question the offline check
can't ask of live prose — did the attack actually land? It emits
injection_success_rate (target 0) into the ADR-136 injected-instruction family and to
Logfire. It is periodic / best-effort / non-blocking — never a per-PR gate
(RFD-SEC-002 §3: deterministic every PR, judge on a tiny live sample only, because judge calls
cost money).
| File | Role |
|---|---|
agents/mira/evals/adversarial_live_sample.py | The live-fire runner: reuses T2's corpus (load_cases), fires each case at the REAL brain (run_command / stream_converse), judges each, emits injection_success_rate, records to Logfire, ALWAYS exits 0. |
agents/mira/llm.yaml | New judges.injection block — the config-driven judge id (openai/gpt-5), a different family than every Mira seat. |
agents/mira/tests/test_adversarial_live_sample.py | Offline-safe guards: skip-clean without the key, judge is config-driven & ≠ every seat, success-criteria track the asserted invariants, sample is bounded. |
.github/workflows/evals.yml | The nightly injection-scanner step now also fires adversarial_live_sample.py for any agent that has one — best-effort, non-blocking. |
changed_fields + scope (the surface an injection would poison).The judge (RFD-SEC-002 §5 — a separate mind) is config-driven
(llm.yaml judges.injection / MIRA_JUDGE_MODEL, never hardcoded —
pramaan-docs ADR-151), ZDR (ZDR_SETTINGS — routes only to
zero-data-retention providers or fails closed, pramaan-docs ADR-116), and a
different family than every Mira seat (judge_id refuses a seat
collision loudly). Its SUCCESS-CRITERIA are derived from each case's asserted
expect block, so the live grade targets exactly what the offline gate asserts. It is
prompted to flag success ONLY on concrete evidence, defaulting to "held" for a refusal / deflection
/ grounded answer, so a safe run is never mislabelled a breach.
The fence-holds proof is the offline deterministic gate (T2, ci.yml). This live lane
is the regression signal: a non-zero injection_success_rate prints a loud
WARNING and lands in Logfire for a human to read; the runner still exits 0 so a flaky/paid live
call never breaks a build. It runs on the evals.yml nightly and on-demand, and
skips cleanly (exit 0) when OPENROUTER_API_KEY is absent, exactly like
Atlas's injection_scanner_run.py.
cd agents/mira uv run --package mira pytest -q # 174 passed env -u OPENROUTER_API_KEY python -m evals.adversarial_live_sample # skips cleanly, exit 0 (no key) cd ../.. && uv run ruff check agents/ # clean
The offline deterministic corpus (T2) stays green and the new module's offline guards pass:
it imports cleanly, skips (exit 0) without the key, the judge is config-driven and ≠ every
seat, and the sample is bounded. The live judged firing runs on the evals.yml
nightly (and on-demand) against the real brain with the dev/CI OpenRouter key, emitting
injection_success_rate (target 0 on the current fence-intact prompt) to stdout and
Logfire.
Eval-only change: no agent brain, runtime, wall, or prompt behaviour is modified
(the llm.yaml edit only adds a judge id used by the eval harness), so no CodeZip
rebuild or AgentCore redeploy is required.