HTTP root-span attrs + log id remap
When a request misbehaves in production, the only way to find the one trace that matters is to filter on a business fact — which firm, which run, which form. This Story makes the request's root span carry those facts, and makes every log line point back at the exact trace it belongs to.
The mental model: a trace is a parcel moving through the warehouse. The carrier already prints the standard barcode (the trace id). This Story adds the shipping label — firm, run, form — so an operator can search "show me everything for firm 42" instead of reading every parcel. And every paper receipt (log line) now reprints that same barcode so receipts and parcel never drift apart.
Why this exists (the catastrophe it prevents)
Without business dimensions on the span, a stalled run is a needle in a haystack: you have a wall of HTTP server spans keyed by route and status, none of which say whose request it was. Triage degrades to guessing. Without trace ids on the log line — or with the wrong field names leaking through — the logs and the traces live in two separate worlds, and correlating them is manual archaeology.
The fix is two coordinated moves: (1) stamp firm_id/run_id/form_id onto the active request span, and (2) guarantee the structured log line carries trace_id/span_id under PRAMAAN's own field names, never the instrumentor's raw otel* names.
🛰️ Satellite: where the work sits
Per pramaan-docs ADR-104 there is exactly one tracing system — the ADOT Lambda layer owns the TracerProvider. This Story adds nothing that competes with it. It rides the spans the layer already creates and enriches them through the shared helper. No provider is constructed; no X-Ray id generator is touched.
The T1/T2/T3 tiers from pramaan-docs ADR-124 decide who emits what:
| Tier | Meaning | This Story |
|---|---|---|
| T1 | Library-owned plumbing spans (HTTP, DB, AWS, logging) | Enabled by adding the instrumentation packages to the lock so the ADOT exec wrapper auto-loads them. No hand-rolled spans. |
| T2 | Domain attributes on existing spans | firm_id/run_id/form_id via the shared allowlist helper. |
| T3 | Registry-approved manual domain spans | None added. |
✈️ Airplane: the three moving parts
1 · The lock
requirements.in now pins opentelemetry-instrumentation-{fastapi,botocore,sqlalchemy,logging}. The Lambda exec wrapper's opentelemetry-instrument discovers them by entry point (OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=none) and instruments them — the T1 floor — and they import locally and in tests.
2 · The root span
RootSpanAttributeMiddleware in app_factory.py reads the product dimensions after the route resolves and stamps them through set_observability_attributes.
3 · The log line
PramaanLogFormatter surfaces trace_id/span_id, preferring the instrumentor's record values when present and falling back to the live span context — and it drops the raw otel* keys.
🔬 Microscopic: the decisions that matter
Why the dimensions are read after the route resolves
firm_id is not known at the perimeter — it is set as request.state.observability_firm_id by the RLS layer while the request runs. So the middleware enriches in a finally after call_next, when state and routed path params are populated. run_id/form_id reuse the same observability_* state convention, with a path/query fallback. The body is never read, so enrichment cannot disturb the request.
Why the allowlist helper, not a raw set_attribute
Every attribute goes through set_observability_attributes, the shared helper from the FN-HELP work. It rejects any key that is not a current OTel semconv key or an approved PRAMAAN product dimension, and rejects None. That is why unrelated request state can never leak onto a span, and why firm_id stays a span attribute and is never placed in baggage.
Why the middleware is innermost — and why the order test changed
It is added first so it sits closest to the route: ServiceIdentity → CorrelationId → RequestCompletion → PayloadLogging → RootSpanAttribute. The pinned middleware-order test was updated to assert this fifth entry. The order is a deliberate contract (ADR-065), so the test pins it on purpose.
Why the otel* keys are remapped and dropped
opentelemetry-instrumentation-logging injects otelTraceID/otelSpanID onto records. Left alone they would appear in the JSON under those raw names — a second, non-PRAMAAN spelling of the same fact. The formatter reads them, emits them as trace_id/span_id, treats the instrumentor's all-zero "no active span" sentinel as absent (falling back to the live span context), and excludes the otel* names from extras so they never leak.
Semconv check (standard vs custom keys)
The semconv guardrail (pramaan-spec#1400) requires current standard keys for standard telemetry and custom keys only for product dimensions.
- Custom keys emitted by this code:
firm_id,run_id,form_id— all approved PRAMAAN product dimensions, gated by the shared allowlist. - Standard keys: none are emitted by this code directly. The HTTP server span's standard attributes (
http.request.method,http.route,http.response.status_code) are owned byopentelemetry-instrumentation-fastapiunder current OTel semconv; this Story only adds the package, it does not hand-stamp those keys.
How to verify
uv run ruff checkanduv run mypy src/shared/app_factory.py src/shared/observability/logger.py— clean.pytest tests/unit/shared/test_root_span_attributes.py tests/unit/shared/test_app_factory.py tests/unit/shared/observability/test_logger_trace_fields.py— the span receives only allowlisted product keys; the log line carriestrace_id/span_idand never theotel*names..github/scripts/observability-guardrail-check.sh— passes (no raw tracer calls outside helper files).- OpenAPI regenerated — no drift, because no route or DTO changed.
Engineering outcome: an operator can now filter a trace search by firm, run, or form, and every log line links to the exact trace — without adding a second tracing path or competing with the ADOT-owned provider.