Story 1499: document_status_events — append-only transitions + observability proof gate

Why this exists (the catastrophe it prevents). A litigation document moves through a processing lifecycle — created, uploaded, and later (EPIC-009) extracting, classified, failed, reprocessing. The documents.documents row holds only the current status; it cannot say how the file got there. Without a recorded history, the matter-detail drawer has no timeline to render, and — worse — when a single upload misbehaves in production, an operator has no thread tying the API request to the work that followed. Custody you cannot reconstruct is not custody. This story adds the append-only ledger that records every transition and stamps each one with the request's trace_id, so the timeline exists and one upload is provably one trace.

The shape: one append-only table, one emit home

A new firm-scoped, append-only table documents.document_status_events holds one row per processing-status transition. It is the operational history beside the document's current-status truth — deliberately separate from ops.audit_log (the firm-level compliance record), because the two answer different questions and have different lifetimes.

presign-upload  -> Document.status = 'pending'   -> record(None      -> 'pending')
finalize        -> Document.status = 'uploaded'  -> record('pending' -> 'uploaded')
(EPIC-009 worker) -> 'extracting' / 'failed' ... -> record(...) via the SAME helper

Every transition flows through one method, DocumentStatusEventService.record_status_transition, so no call site has to remember to capture the trace or stamp the firm. New transitions added by the Mode-2/3 worker pipeline emit through the identical path.

ColumnMeaning / Mode discipline
document_id, firm_idFK to documents.documents (ON DELETE CASCADE) and business.firms. The timeline is the document's; erasing the document erases its timeline.
from_status / to_statusfrom_status is null only for the first transition into the lifecycle; every later row records where it came from.
actor_type / actor_id'user' | 'agent' | 'system' (CHECK-constrained). Mode-1 transitions are user; the worker will stamp agent/system.
job_idNullable now; the EPIC-009 processing job stamps it later — no migration needed (Mode discipline).
trace_id32-char hex of the active span, CHECK ~ '^[0-9a-f]{32}$'. Nullable: tracing can be disabled (OTEL_SDK_DISABLED).
errorFailure detail for transitions into a failed state; null otherwise.

Tenancy & tamper-evidence

RLS mirrors documents.documents (0031): firm_isolation_policy (firm members see only their firm) plus staff_with_grant_policy (staff read only with a live ops.firm_support_grants row). Append-only is enforced at the privilege layer exactly like agents.agent_event (0061): the app role is granted SELECT, INSERT and has UPDATE, DELETE revoked. The new table is registered in the support-confidentiality list so on-duty support (config scope, no content grant) sees zero rows.

Observability: the proof gate

The acceptance criterion is that one upload shows the same trace_id across API → event → audit_log. The emit helper reads the trace from the active span via the single approved reader get_current_trace_id() — it does not open its own span. Per pramaan-docs ADR-124, the manual-span registry is closed and has no document status-event name, and a small DB write inside the caller's request span needs no span of its own; it is covered by the FastAPI route span (the worker consumer span later). The row records the trace so it is queryable back to that span. No raw tracer calls; no file content on attributes.

record_status_transition(...):
    event.trace_id = get_current_trace_id()   # active span, never caller-supplied
    session.add(event); await session.flush() # inside the caller's transaction

Because the write lives in the caller's open transaction, a later failure rolls the event back with the work — the ledger never records a transition that did not commit.

Reading the timeline

GET /app/documents/{doc_id}/status-events returns the transitions newest-first (operationId: appListDocumentStatusEvents, response_model=DocumentStatusEventListResponse, canonical ErrorResponse on every error code). It 404s for an unknown or cross-tenant document id (RLS confines the existence check to the caller's firm), so the two are indistinguishable to the caller.

Acceptance evidence

Acceptance criterionTest / check
Table + migration (up/down clean, metadata parity)make test-db-alembic-roundtrip → alembic check: “No new upgrade operations detected”
Every transition emits a row (presign + finalize), listed newest-firsttest_status_events_recorded_through_router_and_listed
Observability proof: one upload's events carry the active span's trace_id; audit rows co-exist under that tracetest_status_events_capture_active_trace_id
Firm isolation (cross-tenant timeline 404; firm-A sees zero firm-B events)test_status_events_cross_tenant_isolation
Support confidentiality registers the new tabletest_support_app_session_reads_no_confidential_tables
OpenAPI snapshots regenerated, canonical error responsesmake openapi + test_openapi_operations_declare_canonical_error_responses

Review gate — can a transition escape the ledger, or leak across tenants? No. Both Mode-1 transitions (and every future worker transition) write through the one record_status_transition home inside the caller's transaction, so an emit is atomic with the work and impossible to forget. RLS + FORCE confines reads and writes to the firm; the app role cannot UPDATE/DELETE the table; support without a grant sees nothing. trace_id is read from the active span, never trusted from the caller.

Recall: Story 1499 gives every document processing-status transition an append-only, firm-isolated row stamped with the request's trace_id — powering the drawer timeline and proving that one upload is one trace from API to audit log. The outbox → queue → worker links of the full chain arrive with the EPIC-009 document worker; the emit helper is already the shared path those transitions will use.