Story 1464: Non-Fatal Metrics Emission in Request/Worker Paths

Why this exists (the catastrophe it prevents). Story 1445 gave functions registry-backed RED metrics, but it wired emission inside live product paths: the HTTP completion middleware, the Clerk SQS worker, the projection relay, and the WebSocket router all call an emitter on every request or record. Each emitter ultimately calls emit_registered_metric(...), which can raise — from label validation, from the OTel meter, or from the CloudWatch EMF writer. A raised metric error inside those paths turns a perfectly good 200 into a 500, fails an SQS record that already did its real work, or aborts a relay batch. That is the classic inversion: the thing that observes the work breaks the work. pramaan-docs ADR-124 forbids exactly this.

The Fix: One Boundary, Two Speeds

The change adds a single non-fatal boundary around the four public runtime emitters while keeping the strict path intact for developer-time guardrails. Nothing about the metric model (names, units, labels, semconv, histogram design) changed.

caller (middleware / worker / relay / ws router)
  -> emit_http_server_request_duration(...)   <- PUBLIC: never raises
       try:
         build low-cardinality labels
         emit_registered_metric(name, value, labels)   <- STRICT: still raises
       except Exception as error:
         _log_metric_emit_failure(name, error)          <- swallow + warn, then return

Public emitters (emit_http_server_request_duration, emit_sqs_consume_duration, emit_sqs_message_oldest_age, emit_projection_relay_duration) wrap their entire body — label construction included — in try/except Exception. On failure they swallow, log a warning, and return. They never raise.
Strict path (emit_registered_metric and _validate_metric_labels) is untouched and still raises. Tests and developers keep their guardrail: a forbidden label such as firm_id, or any label outside the registry, fails loudly.

What the Warning Logs (and What It Must Not)

_log_metric_emit_failure logs the metric name and the exception class only — never the value, labels, or exception message, any of which could carry tenant or request content. The extra key is metric rather than metric_name because the shared logger redacts any key containing name. The logger.warning call is itself wrapped in contextlib.suppress(Exception) so even a logging failure can never escalate back into the work path.

{"level":"warn","msg":"metric emission failed",
 "metric":"http.server.request.duration","error_type":"RuntimeError"}

Call Sites Confirmed Protected

Every call site reaches the work-path emitters through one of the four public functions, so the single boundary covers all of them. No metric failure can mask or replace the real request/worker outcome.

Call site	Public emitter used
`request_logger.py` success path	`emit_http_server_request_duration`
`request_logger.py` exception path	`emit_http_server_request_duration` (then re-raises the original error)
`clerk_worker/handler.py`	`emit_sqs_consume_duration`, `emit_sqs_message_oldest_age`
`shared/projection_outbox.py`	`emit_projection_relay_duration`
`app_api/router_ws.py`	`emit_http_server_request_duration`

Acceptance Evidence

Acceptance criterion	Test / check
Public HTTP emitter does not raise when OTel recording throws	`test_emit_http_duration_does_not_raise_when_otel_recording_throws`
Public HTTP emitter does not raise when EMF/`single_metric` throws	`test_emit_http_duration_does_not_raise_when_emf_recording_throws`
Middleware still returns the original 200 when metric emission throws	`test_middleware_returns_original_200_when_metric_emission_throws`
Middleware preserves the original exception/status path	`test_middleware_preserves_original_exception_when_metric_emission_throws`
`_validate_metric_labels` still raises for `firm_id`	`test_strict_emit_registered_metric_raises_on_forbidden_firm_id`
Strict path still raises for an unregistered label	`test_strict_emit_registered_metric_raises_on_unregistered_label`
METRIC_SPECS names/units/labels/semconv unchanged	`test_metric_specs_registry_unchanged` + empty `git diff` on the registry block
Walkthrough doc included	this document

Also covered: all four public emitters swallow a fully-down backend (test_all_public_emitters_swallow_backend_failures), and the strict path still propagates a backend failure (test_strict_emit_registered_metric_still_propagates_backend_failure) — proving the boundary lives only at the public surface.

Review gate — can a metrics failure ever break a request/worker path? No. Every work-path call goes through one of the four public emitters, and each public emitter wraps its full body (label build + strict emit) in try/except Exception, swallowing and logging on failure before returning. The warning logger is additionally guarded with contextlib.suppress. The only code that still raises is the strict emit_registered_metric/_validate_metric_labels path, which is reachable only from tests, never from a request or worker.

Recall: Story 1464 makes the four public metric emitters never raise — observability stays loud at dev time (strict validation) but silent-on-failure in production, so a broken meter or EMF writer can never break the request or worker it was measuring.