Story 2064: CE-5 L2 retrieval - hybrid RRF → Cohere Rerank 3.5 → search_evidence

WHY. A legal query needs both legs of recall and neither alone is enough. Full-text search nails the exact-token wins a court turns on - a section number, a case citation, "₹4,20,000" - and is blind to paraphrase. Vector search carries "the respondent denied it" ≈ "the defendant disputed the claim" and is blind to a literal amount. Ship one leg and half the record is unfindable; ship them as two round-trips and the Lambda pays two network hops per search. And raw fused rank is routinely wrong about ORDER, so the top-k a brain actually reads would burn its budget on near-misses. This story fuses both legs in ONE SQL statement, then a reranker fixes the order so the k that reach the brain are the k that matter - and every hit is born with a byte-verifiable SourceSpan, because a retrieval result a brain cannot cite is a fabrication waiting to happen.

The pipeline, built once

search_evidence(matter_id, query, k=12) is three stages and a bounded, near-constant number of statements (RFD-CTX-001 §5.2 / §8): 

query -> embed (Bedrock, search_query)          [not a DB round-trip]
      -> ONE hybrid RRF SQL statement over context.chunks   -> ~50 candidates   [1 DB query]
      -> Cohere Rerank 3.5 (Bedrock, pinned us-east-1)       -> top-k           [not a DB round-trip]
      -> ONE grouped document_pages fetch to materialize spans for the top-k    [1 DB query]
      -> EvidenceHit[] {span, retrieval_score, rerank_score, retrieval in fts|vector|hybrid}

What changed

SurfaceChange
Retrieval executor src/shared/context/retrieval.py The search_evidence executor plus the ONE hybrid-recall statement: two CTEs - vector KNN (embedding <=> :query_vec, HNSW/cosine) and FTS (websearch_to_tsquery('simple', ...), GIN) - both matter-scoped and capped at context_retrieval_candidates, FULL OUTER JOINed on chunk id and fused by Reciprocal Rank Fusion (1/(k+rank) per leg, summed). Firm scoping is the context.chunks RLS policy's job; the SQL only narrows to the matter. Every candidate carries an honest retrieval leg. Bind names only - no value is ever f-strung into SQL.
Reranker adapter src/shared/context/reranker.py Cohere Rerank 3.5 via the Bedrock Rerank API on the bedrock-agent-runtime client (a DIFFERENT endpoint from bedrock-runtime). Same seam CE-3's embedder established: model id and region are constructor args from settings, never hardcoded; boto3 lazy-imported so a fake drops in cleanly; bounded timeouts on the cross-region hop. Fail-closed parsing of the results payload (an out-of-range index is a typed RerankError, not a silently-dropped hit).
Every hit carries a resolvable span A chunk is born from document_pages.blocks, so its (page_start, block_start) envelope resolves straight back to real page bytes. The executor anchors each hit at the chunk's HEAD block and quotes that block's text byte-exactly (a superset that still byte-verifies via CE-2's resolve_span). The block text + segment_kind come from ONE grouped document_pages fetch keyed by (document_id, page) for the reranked top-k - never a query per hit.
Config (settings.py) Three new eval-tunable knobs (ADR-151, tuned by the CE-4 eval, never hardcoded in SQL): CONTEXT_RETRIEVAL_CANDIDATES (default 50), CONTEXT_RRF_K (default 60), CONTEXT_RERANK_MODEL_ID (default cohere.rerank-v3-5:0). The reranker rides the SAME pinned CONTEXT_BEDROCK_REGION (us-east-1) as the embedder - one region for endpoint config, quotas, and debugging.
Provider-call-ban (tests/unit/test_inference_provider_call_ban.py) Extended, not weakened: reranker.py joins embedder.py on the allowlist (the founder-sanctioned Bedrock in-account carve-out for embeddings + rerank only - CE-22 scoped ADR-116/120 amendment; generative inference stays OpenRouter-ZDR), and the ban now also covers the bedrock-agent-runtime client name, so a second Bedrock endpoint cannot leak outside the single adapter seam.
Observability (ADR-124, RFD §14.1) Two registry-listed manual spans wrap the boundary: functions.context.retrieve (INTERNAL - candidate + per-leg counts) around the hybrid SQL, and functions.context.rerank (CLIENT - model id; its duration IS the budgeted cross-region latency) around the rerank hop. The Bedrock client calls and the SQL are T1 auto-instrumented underneath. New low-cardinality attribute keys (retrieval.candidate_count, retrieval.k, per-leg counts) are added to the closed allowlist, structurally excluded from X-Ray annotation promotion.

Query-count discipline (ADR-130)

The whole point of L2 is that a search over a matter with 2 chunks and a matter with 2,000 chunks costs the same number of DB round-trips. search_evidence issues exactly two statements - ONE hybrid RRF recall + ONE grouped page fetch - and never a per-hit query (DR-1/DR-2). The embed and rerank calls are Bedrock, not the database.

Out of scope (named)

Verification (system boundary, not component boundary)

tests/integration/test_context_retrieval.py drives search_evidence against a REAL Postgres (with pgvector) as the runtime pramaan_app role, including the empty first-run state - the real RRF SQL, real FTS, real HNSW, real RLS; only the embedder and reranker are fakes:

Unit tests (test_context_retrieval.py, test_context_reranker.py) cover the SQL shape (one statement, both legs, RRF), reranked-order-wins, fail-closed rerank parsing, and the Bedrock Rerank request shape. The provider-ban, manual-span registry, and observability-guardrail tests stay green with the additive entries. ruff clean; make openapi reports no drift.

One-sentence recall

search_evidence fuses FTS and vector recall in one matter-scoped RRF statement, lets Cohere Rerank 3.5 fix the order to top-k, and hands back hits that each carry both scores and a byte-verifiable span - a bounded two-query, firm-isolated, honestly-empty-safe retrieval seam the whole Context Engine reads through.