Story-2984 Walkthrough

Change: lab sweep now treats cache policy and reviewed compound configurations as real axes. An adapter must report the selected preset and its full resolved configuration; the call-level guard checks that truth immediately before every paid attempt.

Mental model: the axis label is a claim; the effective configuration is the evidence. The sweep may spend only when the claim and evidence match declarations in llm.yaml.

The two new fences

AxisReviewed declarationEffective proof
cache eval_cache_policies.<seat> Every effective seat policy is declared. Runtime None is the canonical representation of reviewed off.
config eval_configs.<seat>.<preset> The adapter reports {preset, values}. The preset name must exist and the complete resolved mapping must equal it after model/cache normalization.

A compound preset cannot hide a cache value inside itself. If it contains a cache mapping, each nested policy is cross-checked against eval_cache_policies, even when the adapter does not separately report the top-level cache axis.

Indigo's preserved comparison is expressible

A_current_uncached_std
  └─ cache off
      ↓ A→B
B_current_cached_std
  └─ cache 5m
      ↓ B→C
C_config1_cached_std
  └─ alternate panel roster
      ↓ C→Z
Z_config1_cached_zdr
  └─ ZDR route

The regression also proves three refusal paths: a preset whose resolved mapping differs from its declaration, an undeclared nested cache policy, and an undeclared preset name. The same selected-value check prevents a table from claiming one variant while running another.

Red-before-green proof

uv run pytest \
  tests/test_lab_sweep.py::test_indigo_cache_and_compound_configs_are_expressible_and_fenced \
  -m unit -q

E   KeyError: 'cache'
FAILED tests/test_lab_sweep.py::test_indigo_cache_and_compound_configs_are_expressible_and_fenced
1 failed in 1.31s

Budget and compatibility proof

The D31 ledger is unchanged: Cartesian preflight, retries, fallbacks, judge calls, per-build spend, and cumulative spend still run through the existing SweepBudget and SweepGuard.

uv run ruff check
→ All checks passed!

uv run mypy evalkit
→ Success: no issues found in 13 source files

uv run pytest tests/ -m unit -q
→ 393 passed, 8 deselected

uv run pytest agents/framer/tests/test_evals_suites.py -m unit -q
→ 4 passed

Not included