Change: CE-16 of the Context Engine (Epic-2049,
RFD pramaan-functions/docs/notes/rfd-context-engine.md §14.3 row CE-16;
§7 EngineState, §8 the tool catalog). CE-15 (Story-2074) wired Mira to
consume the ContextBundle and stamp context_provenance
on turn.completed. CE-16 is the measurement: a non-empty set of
fixture packs (agents/mira/evals/datasets/context/*.yaml) and a Why+Cite grade that
switches its bar on engine_state — the RFD's honesty
mechanism, exercised.
Two clauses drive the grade:
engine_state]”, and “a cited fact whose span is not in the pack
fails closed regardless of engine state.” Canned spans live
only in eval fixtures — never in a production stub.cite(span_id) row): “eval harness grading
Why+Cite mechanically | document_pages byte re-verification.”engine_state | The bar | Why |
|---|---|---|
live (bundle served) | Why + byte-verified Cite | The engine served real record: every produced value must carry a one-sentence producer
rationale (Why, §4) and a source_span.v1 whose quote byte-verifies. |
partial / stubbed / no bundle | frame-only closure | Not-yet-backfilled / stub / frame-only turn — grading degrades gracefully, a Cite is not required (honest empties, never canned content). |
| ANY state, span present | fail-closed fabrication check | A span that does not byte-verify is fabrication and reds the pack even when stubbed. This invariant never degrades. |
cite(span_id)verify_span() is the whole mechanical Cite grade: it resolves a span against the
fixture's canned documents (the offline document_pages) and checks the
quote is byte-present at the anchored coordinates — byte-exact, no normalization
(“a court reads the bytes”, §4). With char_start/char_end
the sliced unit must equal the quote; without them the quote must be byte-present
in the block or page. A stale or forged span returns verified=False →
fabrication.
Five packs, realistic Indian-litigation matter shells, one per honesty state:
| Pack | engine_state | Matter |
|---|---|---|
live-supply-breach | live | Ferro-alloy supply-contract arbitration — GRN + ledger spans |
live-cheque-dishonour | live | s.138 NI Act cheque bounce — return memo + demand notice (page anchor) |
partial-backfill | partial | RFCTLARR land-acquisition compensation — award span, bar dropped |
stubbed-empty | stubbed | Carriage-of-goods stub pre-fetch — honest empty |
no-bundle | — | Demurrage frame-only turn — null-hash provenance marker |
Each pack's golden command runs through the REAL command route (run_command) with
the pack's bundle threaded in, an honest stubbed panel (no key, no egress). The turn's
context_provenance is asserted to mirror the served engine_state, then
the switch grades Why+Cite.
Mirroring the adversarial suite's RED-on-removal proof, test_context_evals.py takes
a real live pack and:
live → RED (the Why+Cite
bar is unmet);stubbed → GREEN (the bar
drops). That contrast is the engine_state switch, proven;live/partial/stubbed →
RED at every state (fail-closed never degrades).cd ~/projects/pramaan-io/pramaan-agents/agents/mira uv run python -m evals.run context # OFFLINE — no key (the blocking CI gate) uv run pytest tests/test_context_evals.py -q
Offline, deterministic, keyless — the shape the RFD grading contract names
(“deterministic invariant checks run on EVERY PR”). A red grade blocks merge
(.github/workflows/ci.yml, the Mira context fixture-pack gate). The live-model /
judge lane over the same packs is the separate, non-blocking sample.
| Criterion | Where |
|---|---|
| fixture-pack eval run green | run.py context — 5 packs held / 0 failures; test_context_evals.py (13 tests) |
| grading switches on engine_state | context.grade_pack — live = Why+Cite, degraded = frame-only closure; both directions proven |
Closes pramaan-io/pramaan-spec#2075.