trace_id through both observability lanes. But the rich genAI detail — model, tokens, tool calls — only ever reached Logfire. CloudWatch's GenAI Observability pane, where AWS-native on-call already lives, saw nothing of the model. Story-1846 fans the brain's genAI spans to CloudWatch too: the same span tree, exported a second time, sanitized of all matter content on the way out.Today agents/atlas/observability/llm_lane.py builds a dedicated (non-global) TracerProvider and points Pydantic-AI at it, with exactly one exporter → Logfire US. That dedicated-provider discipline is the Bug-1294 cure (never own/read the global ADOT setup; never POST unsigned to the SigV4-only platform endpoint). The gap: CloudWatch GenAI Observability ingests any OTel span carrying gen_ai.* semconv — but ours go only to Logfire, so the CloudWatch GenAI pane is empty for an OpenRouter/DeepSeek run.
OTEL_* envs. That is the precise Bug-1294 surface: set_tracer_provider refusal + an unsigned exporter hitting the SigV4 endpoint → repeated 403. The correct move (Spike-1845) is the opposite: keep the dedicated provider, add a second exporter to it.The whole feature is "add one more add_span_processor on the provider we already build." llm_lane.py calls the new hook right after wiring the Logfire exporter, on the same provider object — so a Logfire span and its CloudWatch twin share trace_id/run_id (Story-1843) and span_id.
# agents/atlas/observability/llm_lane.py — the SAME dedicated provider, fanned provider.add_span_processor(BatchSpanProcessor(exporter)) # Logfire US (existing) ... add_cloudwatch_genai_export(provider) # second exporter onto the SAME provider (no new/global one) provider.add_span_processor(AmbientIdentitySpanProcessor())
# agents/atlas/observability/cloudwatch_lane.py def add_cloudwatch_genai_export(provider) -> bool: if not _cloudwatch_genai_export_enabled(): # PRAMAAN_CLOUDWATCH_GENAI_EXPORT not truthy -> no-op return False from amazon.opentelemetry.distro.exporter.otlp.aws.traces.otlp_aws_span_exporter import OTLPAwsSpanExporter session = Session() if session.get_credentials() is None: # flag on but no AWS creds (local/CI) -> clean no-op return False region = _cloudwatch_region() # ap-south-1 xray = OTLPAwsSpanExporter(aws_region=region, session=session, endpoint=f"https://xray.{region}.amazonaws.com/v1/traces") # EXPLICIT (never platform OTEL_* env) provider.add_span_processor(BatchSpanProcessor(D9SanitizingSpanExporter(xray))) return True
set_tracer_provider override; the explicit endpoint means the platform OTEL_EXPORTER_OTLP_* envs are never read; and OTLPAwsSpanExporter SigV4-signs every request — so the old "unsigned → 403" failure is structurally impossible. The two Bug-1294 failure modes are both gone by construction, not by luck.A SpanProcessor runs for every exporter on a provider — putting the scrubber there would also strip Logfire (where, under the synthetic-staging opt-in, content is allowed). So per-lane scrubbing must wrap the exporter. cloudwatch_lane.py's D9SanitizingSpanExporter rebuilds each span content-free just before the AWS call.
# Per-lane D-9 scrub (ADR-146 §5 / ADR-124 D9) — runs in the EXPORT path, never the user path class D9SanitizingSpanExporter(SpanExporter): def export(self, spans): return self._inner.export([_sanitize_span(s) for s in spans]) def _sanitize_span(span): return ReadableSpan( context=span.context, parent=span.parent, # SAME trace_id + span_id -> correlates 1:1 w/ Logfire twin attributes=_sanitize_attributes(span.attributes), # drop gen_ai.input.messages / prompt / completion / tool args events=_sanitize_events(span.events), # drop message/choice content events ...) # name, status, kind, timing, resource, scope, links: verbatim
| What | On the CloudWatch lane | Why |
|---|---|---|
| Prompt / response / tool-payload text | stripped | ADR-124 D9 — no matter content in CloudWatch/X-Ray |
| genAI message / choice events | dropped | the events ARE the content in OTel GenAI semconv |
firm_id (root-span attribute) | kept | tenant pivot; allowed as an attribute, never as baggage (§D10) |
gen_ai.request.model · usage.* · system · run_id · agent.key | kept | exactly what the CloudWatch GenAI pane renders (Spike-1845 §3) |
include_content=False, so spans are content-free in the common case. The sanitizer is the guarantee for the one path that can turn content on — the synthetic/staging opt-in — so a staging misconfiguration can never leak a prompt to CloudWatch.The second exporter rides a BatchSpanProcessor — async, background drain — exactly like the Logfire lane. Never a SimpleSpanProcessor, never a synchronous pre-answer force_flush. The answer streams to the lawyer first; spans flush in the warm tail. The latency invariant from Spike-1844 / Chore-1848 holds with the second exporter ON, and a test pins it structurally so a future edit that moves export onto the answer path fails CI.
# tests/test_atlas_cloudwatch_export.py — the latency invariant, asserted assert isinstance(processor, BatchSpanProcessor) # async, warm-tail flush assert not isinstance(processor, SimpleSpanProcessor) # NOT synchronous on the critical path
Three gates, any one of which keeps the build byte-identical to a CloudWatch-free one — so local dev and CI never reach X-Ray and never need AWS creds:
| Gate | Off → |
|---|---|
PRAMAAN_CLOUDWATCH_GENAI_EXPORT not truthy | no second processor (the default everywhere but staging/prod) |
aws-opentelemetry-distro exporter import fails | caught, logged, no-op (never raises — ADR-110) |
session.get_credentials() is None | no-op (no background 403 spam; the AgentCore role grants X-Ray write only in staging/prod) |
AWSXrayWriteOnlyPolicy to Atlas's execution role is Terraform in pramaan-infra — Story-1849. This repo ships only the code wiring, gated so it is inert until that infra lands. No *.tf in pramaan-agents (AGENTS.md).| Test | Asserts |
|---|---|
d9_sanitizer_strips_content_keeps_firm_and_metadata | prompt/completion/messages gone; firm_id + genAI metadata kept; no privileged text in any value |
d9_sanitizer_drops_content_events_keeps_others | genAI message event dropped, domain event kept |
sanitizer_preserves_span_identity_for_correlation | same trace_id + span_id → 1:1 with the Logfire twin |
dual_export_emits_to_both_sinks_cloudwatch_content_free | one tree, two exporters; Logfire raw, CloudWatch sanitized, matching ids |
flag_absent · flag_on_but_no_credentials | both are clean no-ops (no processor wired) |
flag_on_with_credentials_wires_sigv4_batch_processor | async BatchSpanProcessor wrapping D9 → SigV4 exporter; region + explicit endpoint |
cloudwatch_lane_opens_no_business_infra | thin brain: no DB/SQS/S3 client in the lane (ADR-110 §6) |
627 passed across the Atlas suite (9 new) · ruff clean · brain-egress gate clean.
Doc-citation (Spike-1845 §3) confirms CloudWatch GenAI Observability ingests framework-level OpenRouter genAI spans; what no offline test can show is our specific DeepSeek-via-OpenRouter span rendering in the live console. That residual is converted into a required live K≥3 send-proof artifact, run once the infra (Story-1849) lands — credentials present, Transaction Search on, drive a real Atlas turn, force-flush, find the run_id/trace_id in the aws/spans log group and the GenAI pane. Until then the lane ships gated and inert, with zero rework lost.