Story-3306 Walkthrough

Outcome: a step drop now carries the trace context of the write that produced it, so the clerk worker's functions.step_drop.consume span opens as the CHILD of Blake's agents.registrar.step_drop span. One bundle, one distributed trace, write-by-write — instead of a producer trace and a consumer trace with nothing joining them.

Two repos, and the ORDER matters. The producer field is this PR (pramaan-agents). The extraction is pramaan-functions, and it must be deployed first — see "The ordering is the opposite of the obvious one" below.

🛰️ Satellite — why a shared trace id was not enough

Chore-3304 fixed the producer's own span: it passes the invocation's traceparent explicitly, so the write joins Blake's trace instead of starting a private one. That half is done and is untouched here.

The bytes then leave the process. They land in S3, ring EventBridge, ride SQS, and are read minutes later by a different Lambda. Nothing in that path carried a W3C header, so the consumer parented off the only context it had: the traceparent stored on the agents.agent_run row.

That is a parent, and it is why this is a subtler bug than "two traces". The consumer was in the right TRACE — and still could not answer the question. A clerk run writes THREE drops. Parent all three consumptions off the run and the trace says "the run caused three consumptions" without ever saying which write caused which. The card drop is the one the annexure fan-out is triggered from; you cannot follow it if it looks exactly like the segment drop's consumption.

✈️ Airplane — the hop, before and after

BEFORE                                    AFTER

agent run (broker)                        agent run (broker)
├─ agents.registrar.step_drop  (card)     ├─ agents.registrar.step_drop  (card)
├─ agents.registrar.step_drop  (segment)  │   └─ functions.step_drop.consume   ← the hop
├─ functions.step_drop.consume            └─ agents.registrar.step_drop  (segment)
└─ functions.step_drop.consume                └─ functions.step_drop.consume
   ↑ both hang off the run. Which             ↑ CLIENT → CONSUMER, one pair per write
     write caused which? Unanswerable.

🔬 Microscopic — where it rides, and why not the obvious place

ADR-104 froze traceparent/tracestate as SQS message attributes — the transport's own side channel, never the message body. The S3 equivalent of a message attribute is object metadata (x-amz-meta-*), and result_delivery.py already writes some. So object metadata was the answer the ticket preferred, and the one this Story set out to use.

It cannot work, and that is measured, not argued. Three probes against real S3 in ap-south-1 (full receipts: docs/notes/s3-presigned-put-metadata-2026-08-02.md):

Presigned with	PUT sends	S3 says
Bucket + Key (what the broker mints today)	`x-amz-meta-traceparent`	403 AccessDenied — `There were headers present in the request which were not signed`
Bucket + Key + `Metadata={"traceparent": V}`	the same `V`	200, reads back
Bucket + Key + `Metadata={"traceparent": V}`	a different value	403 SignatureDoesNotMatch

SigV4 signs header values, not just names. An object-metadata header is either signed at MINT time — frozen to what the broker knew before the run started — or refused outright. The value this Story needs is the span id of a write that has not happened yet. There is no version of the metadata channel that can carry it. So it rides step_drop.v1.trace, in the body.

The contract change, stated rather than slipped in

step_drop.v1 is frozen (ADR-180 amendment §2), so the addition is explicit:

The version does NOT bump. trace is optional and additive; no existing field changes meaning and no reader has to reinterpret old bytes. Bumping to v2 would force the consumer's parse_step_drop to refuse every drop until both sides shipped — a flag day for a field that is allowed to be absent.
Absent means exactly one thing: a producer older than Story-3306. The consumer falls back to the run's stored context, which is precisely today's behaviour. An old drop loses the write-level hop; it never loses the trace and is never refused.
The JSON Schema artifact is regenerated (agent_contracts/schemas/step_drop.v1.schema.json) — the ADR-074 custody gate fails the build otherwise.

The ordering is the opposite of the obvious one

The ticket assumed the producer could land first "if the consumer can tolerate an absent field". It cannot: the consumer's ContractModel sets extra="forbid", so a drop carrying an unknown trace field is refused as refused_payload — the whole drop, not just the field.

Consumer first, then producer. Both directions tolerate absence, so the window between the two deploys is safe in one direction only. (The rail is not end-to-end on staging yet, so there is no live traffic in that window today.)

The code

File	Change
`agents/_shared/step_drop.py`	`StepDropV1.trace: TraceContext \| None`. Reuses the artifact contract's existing `TraceContext` — one shape for two strings, not a second one. `StepDropSpanOpener` now yields `(record, carrier)`.
`agents/_shared/artifact_runtime.py`	Reads `traceparent`/`tracestate` off the carrier and stamps them onto the drop before the PUT. Reading exactly two names IS the allowlist.
`agents/registrar/observability/domain_spans.py`	Injects the OPEN span's context with the propagator the module already holds. No new helper: `_shared/` still imports zero OpenTelemetry, which is why the carrier is handed back by the route rather than injected in shared code.

Proof — and the test that fails when the fix is removed

"A test that passes with and without the fix proves nothing", so each claim was checked by deleting the fix and confirming the failure:

Test	Proves	With the stamp removed
`test_each_drop_carries_the_context_of_the_write_that_produced_it`	each drop's `traceparent` carries the run's trace id AND its own write span's id — three drops, three distinct parents	FAILS
`test_only_the_two_w3c_keys_ever_reach_the_wire`	the field holds exactly `traceparent`/`tracestate`; the live presigned URL, document ids and firm id are absent	FAILS
`test_stripping_the_propagated_context_fails_that_same_assertion`	the assertion itself discriminates — run against a route that propagates nothing, it raises	—
pramaan-functions `test_the_consume_span_parents_from_the_write_not_the_run`	the consume span opens under the WRITE's span, not the run's	FAILS

The consumer test earned its keep immediately: it caught that a malformed traceparent was falling through to no parent at all rather than back to the run's context — a garbled field would have cost the whole trace instead of one hop.

Suites: pramaan-agents 484 root + 596 registrar green, ruff + brain-egress gate clean. pramaan-functions: the 13-test step-drop pipeline integration suite green against the real schema on Postgres, plus 9 new unit tests.

What this Story does NOT close — and one thing it found

No live trace link. Acceptance asks for a Logfire link showing the pair on a synthetic bundle. The rail is not end-to-end on staging (the fan-out and ingress halves of EPIC-3286 land separately), and the probe above surfaced a harder blocker:

Every artifact and step-drop PUT will 403 today. result_delivery.py sends x-amz-meta-sha256 and x-amz-meta-size-bytes on every PUT; the broker's _presign_put signs neither. Probe 1 is exactly that request. Signing them does not fix it either — the body's digest is as unknowable at mint time as the span id — while the consumer's verify core REFUSES an object that carries no declared metadata. The two halves cannot both be right.

The fix is a delivery-contract decision (presigned POST, whose policy supports starts-with conditions on client-chosen fields — or the digest in the body), not this Story's. Story-2958's own walkthrough flagged the gap: "Not proved in this repo: a live presigned request must include these exact headers in its signature." It needs its own Bug.

Story-3306's own change is unaffected by that defect and correct either way: the trace context is in the body, which is the one channel a presigned PUT can carry.