Story-1291 stood up the Logfire trace lane (prompt · model · provider · tokens · latency). Bug-1294 then fixed how it coexists with the platform's ADOT layer: instead of logfire.configure() — which seizes the global OpenTelemetry providers and reads the platform's OTEL_EXPORTER_OTLP_* envs (the SigV4-only AWS ingest → repeated 403) — Atlas runs a dedicated, non-global TracerProvider with one explicitly-authed exporter aimed at Logfire US.
The Logfire onboarding checklist has a fourth item that lane was still missing: metrics. Without host/process CPU + memory, a slow or stalled Atlas run shows token spans but no signal on whether the container was starved. The naïve fix — logfire.instrument_system_metrics() — detonates on Bug-1294's exact landmine: it requires logfire.configure(), which would own the global meter provider and pick up the SigV4 ingest, re-introducing the 403 for metrics.
The decision: extend the dedicated-provider discipline to a second signal — a dedicated MeterProvider — and drive the same OTel instrumentor Logfire uses (SystemMetricsInstrumentor) onto it. No global takeover, no behaviour change, no new truth.
cold start (main.py)
│ configure_observability() ← the one hook, called once
▼
LOGFIRE_TOKEN set? ──no──► NO-OP (no provider, no exporter, no egress)
│ yes local dev + CI are byte-identical here
▼
resource = Resource(service.name="atlas", deployment.environment.name=$LOGFIRE_ENVIRONMENT?)
│
├─ TRACE lane (#1291/#1294, unchanged)
│ TracerProvider(resource) + OTLPSpanExporter(/v1/traces, Authorization=token)
│ Agent.instrument_all(tracer_provider=…, event_mode="attributes")
│
└─ METRIC lane (this Story) — independent, non-fatal
MeterProvider(resource, PeriodicExportingMetricReader(
OTLPMetricExporter(/v1/metrics, Authorization=token)))
SystemMetricsInstrumentor().instrument(meter_provider=…)
▼
OTLP egress → Logfire US datacenter project (traces + metrics, shared identity)
One file does the work (observability.py); the cold-start call site (main.py) is unchanged. The brain's request/response contracts, events, and truth boundary stay frozen.
| Piece | What & WHY |
|---|---|
| Dedicated MeterProvider | A second non-global provider built alongside the TracerProvider; meter_provider is passed explicitly to the instrumentor. WHY: the global meter provider belongs to ADOT (general lane, ADR-104); touching it is the Bug-1294 collision. Dedicated = the two lanes never fight. |
Not logfire.instrument_system_metrics() | We call OTel's SystemMetricsInstrumentor directly — exactly what Logfire wraps — pointed at our provider. WHY: the Logfire helper needs logfire.configure(), which seizes the global providers + reads OTEL_EXPORTER_OTLP_* (SigV4 → 403). Same reason #1294 hand-rolled the trace exporter. |
| Explicit US metrics endpoint | OTLPMetricExporter(endpoint="https://logfire-us.pydantic.dev/v1/metrics", headers={"Authorization": token}). WHY: US pin (ADR-118) + auth set in code so no platform OTEL_* env is ever consulted. |
Shared Resource | One Resource (service.name="atlas" + optional deployment.environment.name) feeds both providers. WHY: traces and metrics carry identical service identity + env tag, so they group together in Logfire. |
| Independent + non-fatal | The metrics lane sits in its own try/except after the trace lane. A metrics failure logs a warning and continues — it does not undo the trace lane or take Atlas down. WHY: ADR-110 — observability is strictly best-effort; the brain never blocks the matter. |
| No new PII surface | SystemMetricsInstrumentor with config defaults emits host/process CPU + memory only — no matter content. WHY: the onboarding "safe scrubbing/sampling" item — metrics add zero new sensitive data, so default scrubbing is sufficient. |
| Deps + locks | uv add 'logfire[system-metrics]' → agents/atlas/pyproject.toml (logfire[system-metrics]>=3.0) + uv.lock. Mirrored in repo-root requirements.in; requirements.lock (hashed) regenerated (adds opentelemetry-instrumentation-system-metrics + psutil). |
| Local-dev auth doc | README documents the dev-only interactive CLI auth (logfire --base-url='https://logfire-us.pydantic.dev' auth + projects use --org 'pramaan' 'agents'). WHY: deployed uses LOGFIRE_TOKEN; browser auth must never reach code/CI. |
Traces and metrics are different OTel pipelines (TracerProvider/span exporter vs MeterProvider/metric reader+exporter). The temptation is to let Logfire wire metrics for you — but that one helper drags in logfire.configure() and the whole global-provider problem #1294 already solved. The fix is to repeat the trace lane's exact pattern one signal over: build the provider yourself, auth the exporter yourself, pass the provider in by hand.
provider = TracerProvider(resource=resource)
provider.add_span_processor(
BatchSpanProcessor(OTLPSpanExporter(
endpoint=…"/v1/traces",
headers={"Authorization": token})))
Agent.instrument_all(
InstrumentationSettings(
tracer_provider=provider,
event_mode="attributes"))meter_provider = MeterProvider(
resource=resource,
metric_readers=[PeriodicExportingMetricReader(
OTLPMetricExporter(
endpoint=…"/v1/metrics",
headers={"Authorization": token}))])
SystemMetricsInstrumentor().instrument(
meter_provider=meter_provider)The point: token absent ⇒ nothing changes; token present ⇒ traces + metrics, on dedicated providers only. Tests pin every constraint; the existing Atlas suite stays green.
| Invariant (Story-1308 / ADR-118 constraint) | Proof |
|---|---|
| No token → no-op (metrics lane never touched) | test_no_token_is_a_noop: fakes present yet metric_calls == []; returns False. |
Token → system metrics on a DEDICATED meter provider, US /v1/metrics, token auth | test_system_metrics_wired_on_dedicated_meter_provider: one instrument(meter_provider=…); exporter endpoint = …/v1/metrics, Authorization = token; service.name="atlas". |
| Metrics failure never blocks the trace lane / Atlas | test_metrics_failure_never_blocks_trace_lane: a throwing metric exporter still returns True with the trace lane instrumented once. |
Trace lane (#1294) unchanged — dedicated provider, US /v1/traces, token auth, no global takeover | test_token_wires_dedicated_provider_and_instruments + env-tag + idempotency + token-never-logged tests, all retained and green. |
| Brain doctrine (ADR-110 §6) | Logfire is egress-only — no DB/queue/bucket, no boto3 s3·sqs·dynamodb, no run-truth change. Event-stream / runtime / reflection tests unchanged and green. |
cd agents/atlas && pytest tests -q → 54 passed (51 prior + 3 new). ruff check agents/ → clean. Without LOGFIRE_TOKEN: Atlas behaves byte-identically to before this Story.scripts/deploy-atlas-staging.sh (injects LOGFIRE_TOKEN + LOGFIRE_ENVIRONMENT=staging), then agentcore invoke --target staging. Runtime log: no 403, OpenRouter 200. In Logfire US, filter service atlas · environment staging → confirm system.cpu.* / system.memory.* metrics alongside the Pydantic AI spans.