Change: CE-20 of the Context Engine build (Epic-2049, RFD
pramaan-functions docs/notes/rfd-context-engine.md §5.1 / §14.2, row
CE-20). It adds the card-quality golden eval: it measures the
precision of Blake's DocumentCard on dates / parties /
amounts over the golden corpus — the gate that sits under the L3
assertion & edge writers.
The DocumentCard is the L1 feedstock: L3 "industrializes whatever error rate L1
ships with." A card that invents a date, party, or amount the document never stated
poisons every downstream contradiction and chronology built on it. So the metric the RFD names
is precision: of the facts Blake claims, what fraction are actually in the gold.
Precision, not recall, is the right guard here — a false positive is a graph-poisoning
fabrication; a miss is merely incomplete. This is the sibling of the CE-6 span-reject gate
(card_gate.py): that one asks can every claimed quote be anchored; CE-20
asks is every claimed fact correct.
CE-6/CE-7 already seeded agents/registrar/evals/card_golden_corpus.jsonl: every
line carries a hand-labelled gold card_output (its dates/parties/amounts). CE-20
reuses that corpus whole — it does not fork a second fixture set. Blake is
the card producer, so CE-20 lives beside his card gate in agents/registrar/evals/,
and wires into his eval command set (run.py).
| File | Role |
|---|---|
agents/registrar/evals/card_quality.py | The CE-20 eval. Normalises both the flat gold DTO and the live span-carrying DocumentCard to the SAME three fact-sets (dates by ISO value, parties by casefolded name, amounts by (value_minor, currency)), then computes micro-precision pooled across families. corpus_precision (deterministic identity), perturb_card (the adversary), measure_live_precision (the live/periodic number vs PRECISION_BAR). |
agents/registrar/tests/test_registrar_card_quality.py | The deterministic per-PR gate (no key): identity precision = 1.0, every family is actually labelled, and a perturbed card (one bogus date+party+amount) scores below the bar — the metric has teeth. |
agents/registrar/evals/run.py | Records the live precision snapshot after the classify suite — the "precision bar met and recorded" acceptance. |
agents/registrar/evals/README.md | Documents the CE-6 + CE-20 card-eval pair over the one shared corpus. |
Deterministic, per-PR (CI, no key). Scoring is plain code over the fixtures.
Gold-vs-gold is 1.0 by construction — its job is to red CI the day a contract or
normalisation change silently breaks the scorer, and to prove the metric never false-rejects a
correct extraction. perturb_card appends one fabricated fact per family so the same
corpus, corrupted, falls below PRECISION_BAR — proving the gate is not
vacuous. This is the blocking gate, exactly the shape card_gate.py already uses.
Live, periodic (needs OPENROUTER_API_KEY).
measure_live_precision runs Blake's real card pass over the corpus and
records the true precision against the bar. Sampled/periodic because it costs model calls; it is
not the per-PR gate — same posture as the CE-6 live reject-rate.
| AC | Proof |
|---|---|
| precision bar met and recorded | PRECISION_BAR = 0.90 is named in card_quality.py; measure_live_precision returns passes_bar + the pooled and per-family numbers, and run.py prints them after the classify suite. |
| eval wired as regression | test_registrar_card_quality.py is the deterministic, keyless, per-PR regression (identity=1.0 / perturbed<bar / every family exercised); the live measurement is wired into evals.run. |
uv run --package registrar pytest agents/registrar/tests/test_registrar_card_quality.py -q # 3 passed uv run --package registrar pytest agents/registrar/tests -q # 293 passed uv run ruff check agents/registrar/ # All checks passed!
The per-PR gate is deterministic and offline; the live precision number needs the
key and is recorded periodically via python -m evals.run. Eval-harness-only —
no brain, prompt, llm.yaml, or runtime behaviour change, so no CodeZip rebuild or
AgentCore redeploy is required (Blake stays config+brain only, ADR-110 §6).