Change: the Context Engine's L3 validation layer (RFD-CTX-001 §5.3) needs a
judge-service, not a persona — the assertion normalizer and the
contradiction-edge judge that pramaan-functions CE-9 / CE-10 invoke. This ships that
soul in the agents repo: the prompts, the model roster (llm.yaml,
distinct from Blake's enrichment model), and the refute-stance edge eval
set. It adds NO AgentCore brain — the executor (candidate-blocking SQL, graph writes, run loop)
is functions-side.
agents/context_judge/| File | Role |
|---|---|
llm.yaml | The roster: normalizer (one seat) + judge (a diverse refute-stance panel), on its own CONTEXT_JUDGE_* env namespace — never Blake's REGISTRAR_*. |
config.py | ADR-151 role loader (env override wins, unset fails closed) + family/distinctness helpers. |
prompts.py | The normalizer prompt + the edge-judge prompt, assembled so its REFUTE_CLAUSE is a named, swappable layer. Untrusted (document-derived) assertion text rides inside nonce-walled blocks. |
models.py | Typed I/O — a MINIMAL LOCAL STUB of the CE-1 (assertion.v1/assertion_edge.v1) contracts (not yet merged), plus fact_key and the fail-closed validate_verdict code wall. |
agent.py / provider.py | ZDR-pinned Pydantic AI agent factories over OpenRouter (pramaan-docs ADR-120). |
evals/edge_cases.py + datasets/edges/*.yaml | The refute-stance corpus + deterministic OFFLINE runner — the blocking PR gate. |
evals/live_sample.py + datasets/live/*.yaml | The live judged sample over the roster (non-blocking; the fuzzy, semantic refute calls the offline gate can't make — RFD-SEC-002 §3 grading split). |
tests/ | Roster-distinctness config tests + the refute-stance eval green/RED proof + the code-wall + contract-stub guards. |
1 — refute-stance eval green. The judge is prompted to REFUTE proposed edges:
default to none, emit an edge only on a concrete, span-grounded conflict. The offline
corpus (8 labelled candidate pairs: date/amount/polarity conflicts → CONTRADICTS, corroboration
→ SUPPORTS, an amendment → SUPERSEDES, and different-subject false-pairs → none) fires
through a STUBBED judge and asserts the labels hold. Like Mira's wall proof, it goes RED when
the refute clause is stripped — a stub with no stance turns eager and cries CONTRADICTS on
everything, flipping every SUPPORTS/SUPERSEDES/none case. That RED-on-removal is the proof the eval
tests the refute defence, not nothing.
2 — roster distinctness asserted in config tests. RFD §5.3: "a reader must not grade
his own reading." tests/test_config.py proves, on the shipped llm.yaml:
(a) the judge shares no vendor family with the normalizer (reader ≠ grader); (b) the roster
references none of Blake's REGISTRAR_* enrichment handles and uses its own env namespace
(structural distinctness — it cannot be pinned to Blake's model); (c) the judge is a diverse panel
(≥2 independent families).
prompts.REFUTE_CLAUSE). Rides the fence: flips RED when stripped.models.validate_verdict). An edge citing a span the candidate never carried, or lacking a rationale/confidence, is dropped to none — robust even to a stripped prompt (the Mira scope_held precedent).cd agents/context_judge python -m evals.edge_cases # 8/8 case(s) held / 0 invariant failure(s) (OFFLINE) pytest tests -q # 16 passed cd ../.. && ruff check agents/context_judge/ # All checks passed! python scripts/check_brain_egress.py # clean python scripts/check_prompt_leakage.py # clean pytest tests/ -q -m unit # 170 passed (governance, incl. model-catalog exemption)
Agents-side authoring change: it ships prompts + roster + evals, no AgentCore brain,
runtime, or deploy — no CodeZip rebuild. models.py is a local stub because CE-1
(pramaan-spec #2052) has not landed; when it does, this file becomes a thin re-export (field names
already track the RFD §5.3 registry). The live judged sample (evals/live_sample.py) is
the periodic, non-blocking lane; the offline refute-stance eval is the per-PR gate.