Story 1499: document_status_events — append-only transitions + observability proof gate
Why this exists (the catastrophe it prevents). A litigation document moves through a processing lifecycle — created, uploaded, and later (EPIC-009) extracting, classified, failed, reprocessing. The documents.documents row holds only the current status; it cannot say how the file got there. Without a recorded history, the matter-detail drawer has no timeline to render, and — worse — when a single upload misbehaves in production, an operator has no thread tying the API request to the work that followed. Custody you cannot reconstruct is not custody. This story adds the append-only ledger that records every transition and stamps each one with the request's trace_id, so the timeline exists and one upload is provably one trace.
The shape: one append-only table, one emit home
A new firm-scoped, append-only table documents.document_status_events holds one row per processing-status transition. It is the operational history beside the document's current-status truth — deliberately separate from ops.audit_log (the firm-level compliance record), because the two answer different questions and have different lifetimes.
presign-upload -> Document.status = 'pending' -> record(None -> 'pending')
finalize -> Document.status = 'uploaded' -> record('pending' -> 'uploaded')
(EPIC-009 worker) -> 'extracting' / 'failed' ... -> record(...) via the SAME helper
Every transition flows through one method, DocumentStatusEventService.record_status_transition, so no call site has to remember to capture the trace or stamp the firm. New transitions added by the Mode-2/3 worker pipeline emit through the identical path.
| Column | Meaning / Mode discipline |
|---|---|
document_id, firm_id | FK to documents.documents (ON DELETE CASCADE) and business.firms. The timeline is the document's; erasing the document erases its timeline. |
from_status / to_status | from_status is null only for the first transition into the lifecycle; every later row records where it came from. |
actor_type / actor_id | 'user' | 'agent' | 'system' (CHECK-constrained). Mode-1 transitions are user; the worker will stamp agent/system. |
job_id | Nullable now; the EPIC-009 processing job stamps it later — no migration needed (Mode discipline). |
trace_id | 32-char hex of the active span, CHECK ~ '^[0-9a-f]{32}$'. Nullable: tracing can be disabled (OTEL_SDK_DISABLED). |
error | Failure detail for transitions into a failed state; null otherwise. |
Tenancy & tamper-evidence
RLS mirrors documents.documents (0031): firm_isolation_policy (firm members see only their firm) plus staff_with_grant_policy (staff read only with a live ops.firm_support_grants row). Append-only is enforced at the privilege layer exactly like agents.agent_event (0061): the app role is granted SELECT, INSERT and has UPDATE, DELETE revoked. The new table is registered in the support-confidentiality list so on-duty support (config scope, no content grant) sees zero rows.
Observability: the proof gate
The acceptance criterion is that one upload shows the same trace_id across API → event → audit_log. The emit helper reads the trace from the active span via the single approved reader get_current_trace_id() — it does not open its own span. Per pramaan-docs ADR-124, the manual-span registry is closed and has no document status-event name, and a small DB write inside the caller's request span needs no span of its own; it is covered by the FastAPI route span (the worker consumer span later). The row records the trace so it is queryable back to that span. No raw tracer calls; no file content on attributes.
record_status_transition(...):
event.trace_id = get_current_trace_id() # active span, never caller-supplied
session.add(event); await session.flush() # inside the caller's transaction
Because the write lives in the caller's open transaction, a later failure rolls the event back with the work — the ledger never records a transition that did not commit.
Reading the timeline
GET /app/documents/{doc_id}/status-events returns the transitions newest-first (operationId: appListDocumentStatusEvents, response_model=DocumentStatusEventListResponse, canonical ErrorResponse on every error code). It 404s for an unknown or cross-tenant document id (RLS confines the existence check to the caller's firm), so the two are indistinguishable to the caller.
Acceptance evidence
| Acceptance criterion | Test / check |
|---|---|
| Table + migration (up/down clean, metadata parity) | make test-db-alembic-roundtrip → alembic check: “No new upgrade operations detected” |
| Every transition emits a row (presign + finalize), listed newest-first | test_status_events_recorded_through_router_and_listed |
Observability proof: one upload's events carry the active span's trace_id; audit rows co-exist under that trace | test_status_events_capture_active_trace_id |
| Firm isolation (cross-tenant timeline 404; firm-A sees zero firm-B events) | test_status_events_cross_tenant_isolation |
| Support confidentiality registers the new table | test_support_app_session_reads_no_confidential_tables |
| OpenAPI snapshots regenerated, canonical error responses | make openapi + test_openapi_operations_declare_canonical_error_responses |
Review gate — can a transition escape the ledger, or leak across tenants? No. Both Mode-1 transitions (and every future worker transition) write through the one record_status_transition home inside the caller's transaction, so an emit is atomic with the work and impossible to forget. RLS + FORCE confines reads and writes to the firm; the app role cannot UPDATE/DELETE the table; support without a grant sees nothing. trace_id is read from the active span, never trusted from the caller.
Recall: Story 1499 gives every document processing-status transition an append-only, firm-isolated row stamped with the request's trace_id — powering the drawer timeline and proving that one upload is one trace from API to audit log. The outbox → queue → worker links of the full chain arrive with the EPIC-009 document worker; the emit helper is already the shared path those transitions will use.