Story 1679: Clerk membership webhook observability
Why this exists. Clerk is the source of membership delivery, but Postgres is the source of product authorization. Before this Story, the most important disagreement between those two systems was quiet: if Clerk delivered a membership for a PRAMAAN staff/internal identity, the identity-hat guard returned rejected_identity_hat and wrote a warning log, but no alarmable metric. Unknown Clerk event types were also only logged. Operators could prove what happened after manual log digging, but the system could not page on "Clerk says member, Postgres refused" or on unexpected webhook shapes.
The implemented decision
organizationMembership.created remains the sole membership-creation event. Clerk emits it when an organization invitation is accepted, and the Lexford internal-domain accepts reached the identity-hat guard through that event. A second organizationInvitation.accepted membership path would duplicate work and create a second place to drift.
What changed
| Signal | Behavior |
|---|---|
rejected_identity_hat | The webhook still returns 200 and finalizes the idempotency ledger with rejected_identity_hat, but now emits pramaan.clerk_webhook.problem with outcome rejected_identity_hat. |
| Unknown event type | The route still acknowledges 200 and finalizes ignored_unknown_event, but now emits the same problem metric with bounded event.type=unknown. |
| Handler failure | The transaction still rolls back the ledger row so Clerk can retry, and the route emits the problem metric with outcome failed and a safe error.type. |
organizationInvitation.accepted | Known and intentionally ignored. The route records ignored_known_event, creates no membership row, and emits no problem metric. |
organizationMembership.deleted | The signed route path is covered by a test that verifies the mirrored membership status becomes removed. |
Observability guardrail
The metric uses the approved Python helper path under src/shared/observability/metrics.py. There are no raw tracer calls. Metric labels are bounded: service.name, deployment.environment.name, normalized event.type, outcome, and optional error.type. Exact unknown Clerk event names remain in logs and the ledger, not as metric dimensions.
pramaan.clerk_webhook.problem
type: counter
unit: {event}
alert: any rejected identity hat, unknown event, or handler failure
forbidden labels: firm_id, run_id, matter_id, user_id, trace_id, span_id
Idempotency boundary
The existing ops.processed_webhook_events perimeter ledger remains the idempotency owner. A new Alembic revision extends the outcome CHECK constraint with ignored_known_event, and the SQLModel mirror now includes both rejected_identity_hat and ignored_known_event. Replays still return replay:true without rerunning handlers.
Verification
- Unit metric tests assert the registry-backed EMF shape and nonfatal emitter behavior.
- Route tests assert unknown events emit a problem metric, handler failures emit a problem metric and leave no ledger row, and accepted invitations record
ignored_known_eventwith zero membership rows. - Integration webhook tests assert
rejected_identity_hatreturns the terminal outcome and emits the problem metric. - The signed
organizationMembership.deletedroute path now verifies status flips toremoved.
OPS follow-up. This PR proves local signed-route handling for deletion. Live delivery still needs operator evidence from Clerk Logs and CloudWatch for /aws/lambda/pramaan-app-api / clerk_webhook_* after deploy.