Story 1718: Broker recovery contract (Recovery a/2, Lane functions)
Why this exists (the catastrophe it prevents). Atlas, the Surface Operator, never writes silently: it proposes an operation ("Add a forum called NCLT Mumbai"), counsel confirms, and only then does the broker execute it. That confirm→execute hop is the single most dangerous moment in a legal product, because a network is allowed to lie. A user double-taps Confirm, or the response to the first confirm is lost and the client retries — and now the same filing is created twice. The page the proposal was built on changes underneath it, and the confirm applies the operation to a surface that no longer means what it meant. The write lands but the acknowledgement is dropped, so the system "fails honestly" by retrying — and duplicates the change it already made. A batch of three items half-applies, and the board says "done" while the prose says "failed." Each of these is a wrong-document-in-a-real-matter incident. This Story builds the broker-side recovery contract (RFD-AGT-002 D-10) so none of them can happen: exactly one write per confirmed key, stale proposals rejected before they execute, partial results reported from one source, and an ambiguous outcome resolved by reading — never by re-writing.
agents.agent_idempotency_key claim table — never a parallel idempotency store.
The shape of the change
counsel confirms (echoes the server-minted idempotency_key) | v OperationConfirmation --- idempotency_key (server-minted) + surface_key + contract_version + items | v execute_confirmed_operation(session, firm_id, confirmation, current, executor) | | 1. is_stale(confirmation, current)? --- current = confirm-time re-GET of the live surface | yes -> PramaanConflictError (reject BEFORE any claim or write) | | 2. claim the key in agent_idempotency_key (boundary='operation') | INSERT ... ON CONFLICT DO NOTHING | lost the claim? | completed -> replay the stored outcomes (executor NOT re-run) | uncompleted-> ambiguous: resolve by GET, never re-write | | 3. run executor(item) per item --- one failure does not sink the batch | | 4. persist outcomes + mark complete -> return ONE OperationResult v OperationResult --- outcomes[] is the single source; board AND prose read it
What this ships
1. The recovery contract (pure) — src/shared/agent_contracts/operation_recovery.py
A new operation_recovery.v1 wire contract, kept pure (no sqlalchemy/boto3/settings) so pramaan-agents imports it as the single source of truth (ADR-110 §5). It defines the confirm envelope (OperationConfirmation — the idempotency_key is a UUID, surface_key is URL-rejected, items is non-empty), the confirm-time re-read (CurrentSurfaceState), the pure staleness predicate (is_stale), and the partial-result object (OperationResult over per-item OperationItemOutcomes). The four outcome states are closed: applied / failed / skipped / ambiguous.
2. The broker (DB) — src/shared/agent_operation_broker.py
execute_confirmed_operation enforces all four D-10 guarantees against the live database inside the caller's RLS-scoped transaction. mint_operation_idempotency_key() is the server-side minting point — the key is never client-chosen, so a client cannot forge or collide keys. The claim reuses agents.agent_idempotency_key under a new operation boundary, mirroring how Blake's registration boundary reuses the same table.
3. The idempotency boundary — migration 0129 + agents_db_models.py
The operation boundary joins the valid-value CHECK and the per-boundary shape CHECK (its shape: the server-minted key rides request_id; step/tool/sha unused). A partial unique index (firm_id, request_id) WHERE boundary='operation' is what makes "exactly once" true: a double-confirm collides on INSERT, so the claim's ON CONFLICT DO NOTHING loses and the second confirm replays instead of re-writing.
The decisions that matter
Server mints the key; the client only echoes it
If the client chose the idempotency key, a buggy or hostile client could collide two distinct operations onto one key (silently dropping the second write) or spray fresh keys to defeat dedup entirely. Minting server-side at propose-time removes both: the key is opaque to the client, unique by construction, and the unique index is the hard backstop.
Reject staleness before claiming or writing
The broker does not trust the contract_version the client echoed — it takes a confirm-time CurrentSurfaceState (a re-GET of the live surface contract) and compares. A drift on either axis (the surface changed, or its capabilities changed) means the proposal is stale, and the operation is rejected with pramaan.agent_operation.stale before any claim row or write exists. Nothing to undo, because nothing happened.
Ambiguous ≠ failed: resolve by a GET, never a silent re-write
A claim that was won but never completed is the applied-but-unacked case — a prior confirm crashed between claiming and finishing. Treating that as "failed→retry" is exactly how you file the same motion twice. The broker instead returns an ambiguous result carrying a single recovery instruction (reload to check; do not re-confirm) and never re-runs the executor. A GET resolves the truth; the broker never gambles a duplicate.
One result object, so the board and the prose cannot disagree
OperationResult.outcomes is the only source. The board view (the per-state tally()) and any prose summary are both derived from that one list; nothing stores a second count that could drift. On a replay, only the outcomes are read back from the claim row — the roll-up is recomputed, never persisted — so a replayed result reports identically to the first run.
Proving it at the real boundary
INSERT ... ON CONFLICT against a real unique index. So the proof runs as the pramaan_app role under RLS against a freshly provisioned Postgres, including the empty first-run state where no prior claim row exists. The five integration cases: a double-confirm of one key runs the executor exactly once and the second confirm replays (replayed=True, executor call count stays 1, one claim row); a stale confirmation raises before any claim row is written and the executor never runs; a claimed-but-uncompleted key (a simulated crash) resolves to an ambiguous GET-recovery result with the executor never called; a partial batch with one failing item reports applied=1, failed=1; and firm B cannot see firm A's claim row yet may claim the same key string as its own first-run operation (RLS isolation).| Confirm | Broker outcome |
|---|---|
| first confirm, fresh surface | claim won → executor runs per item → outcomes persisted → OperationResult (replayed=false) |
| second confirm, same key | claim lost, row complete → replay stored outcomes; executor never re-runs (exactly once) |
| confirm against a drifted surface/version | pramaan.agent_operation.stale — rejected before any claim or write |
| confirm, prior claim won but never completed | ambiguous result + GET-recovery hint; never re-writes |
| batch, one item's write fails | that item failed, the rest applied; one result object |
Verification
ruff checkclean on all changed files.- mypy clean on the two new modules (the only mypy noise is a pre-existing missing-yaml-stub in
risk_policy.py; mypy is local-only). - Full unit suite green (1061 passed), including the new contract/helper/migration tests and the updated boundary-set assertions in the 0118/0065 migration tests.
- The five DB-backed recovery cases pass as
pramaan_appunder RLS against the isolated test Postgres. - Migration 0129 proven against real Postgres:
upgrade head,downgrade -1 → upgrade headroundtrip clean, andalembic checkreports no drift. make openapi— no drift (operation_recovery.v1is a snake_case broker/brain wire contract, not an OpenAPI DTO surface).- ADR-124: the broker's small claim writes ride the caller's broker span (the closed manual-span registry has no operation-broker name, the same documented exemption the effect ledger uses); only low-cardinality product IDs + the active
trace_idare logged, never params, neverfirm_idin baggage.
Sibling close gate (RFD D-10, not in this PR): the counsel-facing fail-honest phrasing, the undo-as-a-write intent, and the never-fake-success behavior are the pramaan-agents Recovery (b/2) child. This PR delivers the broker harness those behaviors execute against.