Decommission to one tracing system + ADR-104 trace bridge [Story-1138]

Story 1138Epic None35 files · +499/−2006

PR #290 · pramaan-io/pramaan-functions · merged 2026-06-10

Why this matters

Two tracing stacks. One request. Zero trust between them.

Custom FastAPI instrumentation, monkey-patched services and Powertools Tracer all writing the same request — cold-starts balloon, span IDs don't correlate, logs lose their traces.

More wrappers = more cost, less clarity.

The fix — One tracking number follows every parcel — duplicate tracing ends at SQS, not at the dashboard.

How it flows

sequenceDiagram
  autonumber
  participant client as Client
  participant lambda as Lambda (Powertools tracer)
  participant sqs as SQS
  participant adot as ADOT :22
  participant logs as Logs Insights
  client->>lambda: traceparent header
  lambda->>lambda: tracer.capture_lambda_handler → root span
  lambda->>sqs: msg attrs: traceparent, tracestate, correlation_id
  sqs->>lambda: next handler resumes same trace
  lambda-->>adot: xray + tracecontext export
  adot-->>logs: Logs Insights joins trace_id + correlation_id

requirements.lock · +38 — new methods

Methodin → outwhat it does
_trace_message_attributes() → dict[str, Any]

What changed, file by file

Adds `_trace_message_attributes` so traceparent, tracestate and correlation_id travel on every SQS message; the trace survives the queue hop (claim: SQS enqueue carries those attributes).

new methodsignature
_trace_message_attributes() → dict[str, Any]

Adds an optional `correlation_id` on `AgentCommand` so older frontends that omit it still produce valid commands (claim: AgentCommand.correlation_id optional + backward compatible).

New fields: correlation_id : str | None

Files changed (35)

File±
.github/workflows/deploy-admin-api.yml+5 −6
.github/workflows/deploy-app-api.yml+5 −6
.github/workflows/deploy-ingest-api.yml+4 −5
.github/workflows/deploy-layer.yml+2 −3
.github/workflows/provision-agents-infra.yml+6 −7
AGENTS.md ★+18
docs/walkthroughs/story-1138-walkthrough.html+126
openapi/admin-api.json+0 −277
requirements.in+3 −4
requirements.lock ★+38 −133
src/admin_api/app.py+0 −2
src/app_api/router_ws.py ★+29 −6
src/domains/observability/__init__.py+0 −1
src/domains/observability/dtos.py+0 −76
src/domains/observability/exceptions.py+0 −19
src/domains/observability/trace_service.py+0 −232
src/shared/agent_commands.py+7
src/shared/app_factory.py+1 −23
src/shared/aws/__init__.py+0 −1
src/shared/aws/cloudwatch_logs.py+0 −120
src/shared/db/session.py+0 −2
src/shared/observability/logger.py+5
src/shared/observability/middleware.py+6
src/shared/observability/tracing.py+0 −271
src/shared/routes/traces.py+0 −72
src/shared/settings.py+0 −22
src/shared/utils/observability.py+0 −20
tests/integration/test_admin_traces_route.py+0 −193
tests/routes/test_admin_traces.py+0 −191
tests/unit/observability/test_trace_service.py+0 −235
tests/unit/shared/observability/test_logger_trace_fields.py+53
tests/unit/shared/observability/test_trace_bridge.py+94
tests/unit/shared/observability/test_tracing.py+0 −79
tests/unit/test_agent_commands.py+26
tests/unit/test_router_ws.py+71