Story 522: Upload endpoint — presigned PUT, malware scan gate, server-computed hash & MIME

Why this exists (the catastrophe it prevents). A litigation matter's documents are evidence. Two failures are unacceptable. First, a malware-laden file reaching the extraction/OCR pipeline (or being served back to a lawyer) — the upload surface is an attacker's door into the firm. Second, a file that the system claims is some document but whose bytes were swapped, truncated, or never actually arrived — evidence whose integrity cannot be proven is evidence a court can throw out. Story-522 closes both: every uploaded object is malware-scanned before it can be extracted or served, and the system computes its own SHA-256 and MIME from the real S3 bytes rather than trusting anything the client said.

The lifecycle (ADR-126): row before bytes, scan before extract

The documents.documents control row drives an explicit state machine. Story-522 owns the column reshape (status → processing_status, widened) and moves rows up to stored; the extraction worker (#523/#524) drives stored onward.

pending ──finalize (HEAD-verify + 100MB cap)──▶ uploaded
   uploaded ──GuardDuty scan result──┬─ clean ─▶ stored ──▶ (extracting ─▶ indexed)  [#523/#524]
                                     ├─ threat ─▶ failed        (quarantine)
                                     └─ unscannable ─▶ needs_review

A document is usable (downloadable, copyable, evidence-eligible) once it is malware-clean: uploaded and every state past it. failed / needs_review are gated out of every usable surface, including the FN-PRIVILEGE evidence-eligibility index.

POST upload-url → the presigned PUT

presign_upload creates the row at processing_status='pending' (row-before-bytes) and returns a presigned S3 PUT to the ADR-126 D2 key shape:

firms/{firm}/matters/{matter}/documents/{document}/raw/{safe-filename}

The raw/ segment is the WORM source half of the raw/derived durability split; cross-matter copies (FN-LINEAGE) inherit the same shape, so the copied source also lands under raw/. The key embeds the firm and document ids — this is what later lets the scan consumer resolve the row from a GuardDuty message that only names the S3 object.

POST upload-complete (finalize) → verify, cap, no extract

Because the row exists before the bytes, an abandoned PUT would otherwise leave a phantom pending row claiming a file that never arrived. finalize_upload proves the object with HeadObject — authoritative size, not the client's claim — enforces the 100 MB server cap (a presigned PUT cannot bound the body, so the ceiling is checked here), and rejects a declared size that disagrees with reality. It transitions to uploaded and stamps the audit row. It deliberately enqueues no extraction: the object is present but not yet scanned.

The malware scan gate (second consumer)

GuardDuty Malware Protection for S3 scans every object on write and publishes a result to pramaan-documents-malware-results-staging. A dedicated SQS-triggered Lambda (src/workers/document_malware_results) drains that queue and crosses the gate via DocumentService.apply_scan_result:

clean        ─▶ re-hash the real S3 bytes into checksum_sha256,
                  detect mime_type from magic bytes, set firm dedup edge,
                  status uploaded ─▶ stored, THEN enqueue extraction (same txn)
threats_found ─▶ status uploaded ─▶ failed   (quarantine; never extracted)
unscannable  ─▶ status uploaded ─▶ needs_review   (human triage)

Tenancy: a system worker that honors RLS

GuardDuty's message names only the S3 object, not our firm. The worker parses (firm_id, document_id) out of the managed key and sets the firm RLS context. The documents policy is business.can_read_matter, which grants firm_admin/firm_editor full firm access — so the worker acts as firm_admin within that one firm. RLS still confines every statement to it (can_read_matter requires firm_id = current_firm_id); there is no BYPASSRLS.

Firm-scoped dedup

At the stored transition the worker probes for the earliest scan-clean document in the same firm carrying the same checksum_sha256 and records it in duplicate_of (self-FK, RESTRICT, firm-leading partial index). The same file uploaded twice is two rows, one dedup edge — the duplicate is identified without being merged or hidden.

Observability (pramaan-docs ADR-124)

The SQS consumer and its DB writes sit on the ADOT T1 auto-instrumentation floor (Lambda invocation span + SQS/DB spans), exactly like the documents domain's existing precedent (copy, retention) — the closed manual-span registry names no malware-result boundary, so none is hand-rolled. The stored transition's trace_id and the extraction outbox's traceparent are both stamped from the live span, so the trace stays continuous into the extraction worker.

Acceptance evidence