Change: RFD-AGT-010 W1/T1 (§4 critic-gate invariant, D3/D4; blake
one-pager §3.5 D7/D8). Blake's HITL auto-doctrine flips classify fields
(display_name/document_type/tags/filing_relation/description)
toward auto. Auto with no human backstop REQUIRES a machine gate.
Blake already had a panel — but it was govern-only (govern.aggregate_panel,
the 3-lens vocabulary vote). The classify path had no critic. This story adds one,
the same shape as the govern panel, and wires its verdict into the tool-policy auto-rule.
Mirrors the govern gate exactly. A distinct-roster model audits Blake's
classify proposal against the anchored evidence and emits a flat
ClassifyCriticVerdict (the "evidence accountant" — every endorsed value must
trace to the quoted document; a value it can't trace is a fabrication, a smuggled instruction is
an injection). Then code, not the model's word, decides pass/not-passed:
verdict=="pass" ∧ no fabrication ∧ no injection ∧ no unsupported field ∧ conf ≥ bar → PASSED anything else (fail / abstain / absent / below-bar / blocked field) → NOT PASSED
Default is NOT-passed (reader ≠ judge). A silent or failing critic never
earns auto — the sibling of govern.aggregate_panel's
default-reject. The bar (critic_pass_min) lives in registrar.yaml, a
reviewed config change, never code.
The critic roster shares NO model family with the classify roster — a
judge grading its own family proves nothing (mirrors Vega's judge-seat-clash guard). The roster
is config-driven in model-catalog.yaml critic_roster (ADR-151, env
handles, never hardcoded); config.assert_distinct_roster computes vendor families
(the part before /) and raises RosterClashError if the critic overlaps
the classify roster. A config edit that accidentally reuses a family reddens CI, never
self-grades silently.
A classify field reaches auto ONLY when both hold: (a) the critic
passed it, AND (b) its deterministic eval gate is green. This extends the existing gate-rule:
GateEvidence gains a critic_passed set, and
auto_eligible adds one clause — a classify-field tool not in
critic_passed is never eligible, however green its gates. Absent/failing critic
→ the field stays suggest. This story builds the GATE, not the
flip — nothing is flipped to auto; the runtime intent-emission path
is untouched.
| File | Role |
|---|---|
agents/registrar/models.py | The flat ClassifyCriticVerdict (verdict / confidence / fabrication / injection / unsupported_fields / findings) — same shape family as a govern lens verdict, PromptedOutput-safe. |
agents/registrar/critic.py | The deterministic aggregator: aggregate_critic(verdict) → CriticDecision(passed, ...) — default-reject, sibling of govern.aggregate_panel. Pure; reads the bar from registrar.yaml. |
agents/registrar/prompts/critic.py | The critic system prompt (PROMPTING.md: evidence-accountant, recursive injection fence, trace-every-value, default-not-pass) + the audit user-prompt builder. |
agents/registrar/config.py, model-catalog.yaml | critic_roster (distinct-vendor env handles) + load_critic_roster / assert_distinct_roster / build_critic_config — the fail-loud reader≠judge guard. |
agents/registrar/agent.py | _classify_critic_agent + run_classify_critic — the distinct-roster pass; fail-CLOSED to abstain on a run error, fail-LOUD on a roster clash. |
agents/registrar/evals/gate_rule.py | GateEvidence.critic_passed + the auto_eligible clause + _CLASSIFY_CRITIC_TOOLS — the wiring. |
agents/registrar/evals/critic_dataset.py, evals/critic_cases.jsonl | The offline gate suite: canned critic verdicts (the model frozen, govern-suite discipline) + the labelled corpus + the ClassifyCriticCatchesFabrication invariant. |
agents/registrar/registrar.yaml, manual_span.py | The normative classify_critic: block + confidence.critic_pass_min; the agents.registrar.classify_critic domain span (ADR-124). |
The critic's model part is the verdict; the part the auto-gate actually guards is the
aggregator. So each fixture freezes its verdict as data and the Task is the
aggregator — no model, no key, green in CI on every push (exactly how the govern suite
freezes lens verdicts). The live critic is a periodic sample (evals/run.py),
never the per-PR gate.
| AC | Proof |
|---|---|
| a classify-critic reviews the proposal; distinct roster (enforced) | test_registrar_critic.py — build_critic_config builds on a distinct roster and raises RosterClashError on a family overlap; run_classify_critic runs the distinct-roster agent. |
a field reaches auto ONLY with critic pass + green gate; else suggest | test_classify_field_needs_a_critic_pass_to_reach_auto — assign_document_type with every gate green + bar met but no critic → auto_eligible False; with the critic pass → True. A non-classify auto tool is unaffected. |
| a fabricated/injected classify value is caught on the labelled set | test_no_fabricated_or_injected_case_ever_passes over critic_cases.jsonl — every fabricated / injected / unsupported / abstain / below-bar case is NEVER passed (so never reaches auto); only clean passes. |
registrar eval + governance suites green; ruff clean | See Verification. |
uv run --package registrar pytest tests -q # 284 passed uv run ruff check agents/ # All checks passed! python scripts/check_brain_egress.py # brain-egress check: clean python scripts/check_prompt_leakage.py # prompt-leakage check: clean uv run pytest tests/ -q # 175 passed (governance)
Scope guard: registration behaviour, the span gate, and intent emission are
UNCHANGED (the runtime classify path is untouched); no field is flipped to auto
(gate only); config families stay in Blake's catalog; the critic model is never hardcoded. Brain
stays config+brain only (pramaan-docs ADR-110 §6): the critic holds no DB/DSN, calls no
tool, and its only egress is the inference provider (distinct roster, ZDR-pinned).