Story 2053: the embedding retrieval eval (CE-4)

Why this exists (the catastrophe it prevents)

The whole Context Engine stands on ONE bet: that when an agent asks "where does the record discuss the minimum offtake obligation?", the vector index hands back the right chunks. If the embedding model is wrong for Indian legal prose, every layer above it - the reranker, the bundle, the Why/Cite chain, the contradiction graph - is confidently built on garbage retrieval, and nobody notices until counsel is staring at a fabricated citation. The RFD's own kill criterion (§3) says so: if the default model fails the recall bar AND nothing beats it, stop and rethink retrieval before the bundle ships evidence.

So the model choice cannot be a vibe. It needs a golden Indian-legal corpus and a repeatable score. That is this story: a committed regression harness that measures recall@k for the three RFD candidates and, as an explicit second output, sizes the per-surface context budgets. cohere.embed-v4 is the decided default (founder, 2026-07-06) and ships unless clearly beaten - the harness is the thing that would catch a "clearly beaten".

Satellite: what the harness produces

tests/evals/context/
  datasets/corpus.jsonl    26 Indian-legal chunks: plaint, written statement,
                           agreement clauses, order, invoices, affidavit...
  datasets/queries.jsonl   10 labelled queries (exact-token / paraphrase / anaphora)
  embedders.py             offline lexical stand-in  +  live Bedrock/Voyage adapters
  _harness.py              recall@k under a multi-firm fill  +  budget sweet spot
  run_eval.py              CLI: prints the report, writes VERDICT.md
  test_embedding_eval.py   the regression gate (+ teeth tests)
  VERDICT.md               the recorded verdict

OUTPUT 1  verdict : recall@k per model -> default ships OR swap is triggered
OUTPUT 2  budgets : S / M / L token defaults -> recorded in settings.py

Airplane: one analogy - the multi-firm HNSW fill is a shared filing room

Picture ONE filing room holding every firm's documents in a single index (that is the "one global HNSW graph", critique M-5). When firm Alpha's agent searches, it must only ever see Alpha's matter - a lock on the drawer, not a polite request. In production that lock is RLS + the firm/matter predicate. The harness models it exactly: corpus.jsonl mixes Alpha's target matter with distractor chunks from firm_beta and firm_gamma, and retrieve() applies the predicate before ranking. So the number we report is filtered recall - recall when the model competes against a full, noisy index - not a flattering single-matter benchmark.

          one global pool (all firms)
        +-----------------------------------+
        | alpha/M-1  beta/M-9  gamma/M-2 ... |
        +-----------------------------------+
                     |
          firm+matter predicate  (the drawer lock)
                     v
        +-----------------+
        | alpha/M-1 only  |  <- rank THIS by cosine, measure recall@k
        +-----------------+
   a test asserts a top-scoring beta chunk NEVER crosses the lock

Microscopic: "clearly beaten" is a number, not an opinion

The verdict rule lives in decide_verdict. The default wins unless a candidate beats its recall@5 by at least the CLEAR_WIN_MARGIN (0.05). A margin below that is noise; the decided default ships. A margin at or above it flips swap_triggered, and the rationale names the exact, documented operation - there is no ambiguity about what "a Voyage model won" means operationally:

winner beats default by >= 0.05 recall@5
   =>  swap_triggered = True
   =>  flip CONTEXT_EMBEDDING_MODEL_ID + run the batch re-embed job
       (idempotent on chunk_hash; dimension-migration path if D differs)

Microscopic: the second output - budget sweet spot

§5.5 says the S/M/L tiers survive as names but every number is an eval-tuned config default balancing three forces: latency, cost × expected turns, and lost-in-the-middle quality (past a size, MORE context degrades the answer). budget_sweet_spots sweeps token budgets per tier and picks the utility peak - where the next 1k tokens of quality stops beating its latency + cost. The measured recall curve feeds the lost-in-the-middle plateau check. The result lands at ~5k / ~12k / ~50k and is recorded in settings; consumers map surface to tier: proposer/vega → L, critic/converse → S (synthesizer reuses the proposer pack verbatim, so it carries no budget of its own).

The honesty line: offline validates the machine, live decides the model

What landed where

A note on the missing contract

CE-1 (src/shared/context_contracts/, pramaan-spec#2052) had not landed when this story was built, so the harness uses local dataclasses (Chunk, EvalQuery) instead of the shared DTOs. None of the verdict or budget math depends on the DTO shape; when CE-1 lands the loaders can adopt context.chunks without touching the harness.

Verification

uv run ruff check tests/evals/context/ src/shared/settings.py
uv run pytest tests/evals/context/ -q          # hermetic: no DB, no network, no model
uv run python -m tests.evals.context.run_eval  # print the report
# live bake-off (records the quality verdict):
AWS_PROFILE=... VOYAGE_API_KEY=... uv run python -m tests.evals.context.run_eval --live --write-verdict

Recall in one sentence

Story-2053 is the committed golden-corpus regression that scores embed-v4 against the two Voyage challengers under a realistic multi-firm fill and sizes the per-surface budgets - so "the default ships" is a measured verdict a live run can overturn, never an assumption.