Story 1505: soft-delete + retention + legal-hold (FN-RETENTION)
Why this exists (the catastrophe it prevents). A litigation document — an affidavit, a bank statement, a chargesheet — is evidence. Evidence has two non-negotiable properties that an ordinary file does not: it can be placed under a legal hold that forbids its removal, and its source bytes are stored WORM (write-once-read-many, S3 Object Lock) so they physically cannot be erased on demand (pramaan-docs ADR-126). The naive answer — “delete a document” means DELETE FROM documents and an S3 DeleteObject — is catastrophic here: it would silently destroy WORM-locked evidence, ignore a court’s legal hold, and leave no record that anything ever existed. And “delete a matter” is worse: you cannot hard-drop a matter’s S3 prefix when its raw/ objects are Object-Locked. This Story (roster id FN-RETENTION) makes deletion logical: a soft-delete axis that retains the row and its bytes, a legal hold that blocks deletion, retention-class metadata, and a matter delete that cascades as soft-deletes — never a hard erase.
The mental model: a sealed evidence locker, not a shredder. “Deleting” a document is not feeding it to a shredder. It is moving the folder to a back shelf marked removed — the paper still exists, the chain of custody is intact, and it can be pulled back to the active shelf (restore). A legal hold is a tamper-evident seal on the folder: while the seal is on, no one may move it to the back shelf at all. WORM is the fact that the originals are kept in a vault no one — not even an administrator — can open to burn them before the retention clock runs out.
🛰 Satellite: the whole shape of the change
One migration that adds lifecycle columns to the existing documents.documents table (no new table), one set of service transitions, five routes, two audit verbs. Upload, list, finalize, and download are unchanged except that the default list is now live-only.
availability: live <----restore---- deleted (never DELETE, never S3 erase)
| ^
+----------delete----------+ (blocked if legal_hold = true)
DELETE /app/matters/{id}/documents/{doc} -> soft-delete (409 if held / already deleted)
POST /app/matters/{id}/documents/{doc}/restore -> back to live (409 if not deleted)
POST /app/matters/{id}/documents/{doc}/legal-hold -> place hold (409 if already held)
DELETE /app/matters/{id}/documents/{doc}/legal-hold -> lift hold (409 if not held)
DELETE /app/matters/{id}/documents -> logical matter delete: cascade
soft-delete all live docs
(409 if ANY is under legal hold)
GET /app/matters/{id}/documents?includeDeleted=true -> management view (sees tombstones)
✈️ Airplane: the files that moved
alembic/versions/c2e4a6b8d0f1_0110_documents_retention_legal_hold.py(new) — adds nine columns todocuments.documents:availability(live/deleted),deleted_at,deleted_by_user_id,legal_hold,legal_hold_set_at,legal_hold_set_by_user_id,legal_hold_reason,retention_class(worm_raw/rebuildable_derived),retain_until. Two user FKs (RESTRICT), four check constraints, two partial indexes. Down-revisionf3a1c7e90b24(0109) → this is the new head 0110.src/domains/document/entity.py— the matching nine fields plus the constraints/indexes in__table_args__. Column comments are kept verbatim-identical to the migration soalembic checkreports no drift.src/domains/document/service.py—delete_document,restore_document,set_legal_hold,clear_legal_hold,delete_matter_documents;list_documentsgains a live-only default with aninclude_deletedmanagement escape hatch;_itemnow carries lifecycle fields and a new_lifecycle_viewbuilder.src/domains/document/dtos/document.py—DocumentAvailability/DocumentRetentionClassenums; lifecycle fields onDocumentListItem;DocumentDeleteRequest,DocumentLegalHoldRequest,DocumentLifecycleView,MatterDocumentsDeleteResponse.src/app_api/router_matters.py— the five mutating routes (unique camelCase operationIds), theincludeDeletedquery param on the list route, canonical error responses.src/shared/audit/registry.py— two new access-security verbslegal_hold_placed/legal_hold_lifted; the lifecycle verbsdeleted/restoredandmatter.deletedalready existed.src/shared/audit/coverage.py— the five new mutating routes registered to their audit actions, satisfying the route-coverage watchdog.openapi/app-api.json— regenerated viamake openapi; admin snapshot unchanged (app-only surface).tests/integration/test_app_document_retention.py(new) — soft-delete/restore, legal-hold-blocks-delete, and logical-matter-delete flows, each through the real router against a real RLS DB, asserting the audit trail and that the WORM object key is never touched.
🔬 Microscopic: the decisions inside the change
1. Why columns on documents.documents, not a new table
Soft-delete, legal-hold, and retention are state of a document, not a separate entity with its own identity. Putting them on the existing row keeps the lifecycle atomic with the document, inherits the table’s existing firm-isolation + staff-with-grant RLS for free, and — because documents.documents is already on the support-confidentiality allowlist (test_support_app_session_reads_no_confidential_tables) — introduces no new firm table to register. The app role already holds SELECT/INSERT/UPDATE (never row DELETE) on the table, which is exactly the privilege set logical deletion needs.
2. Why availability never means a hard delete
Deletion flips availability to 'deleted' and stamps deleted_at/deleted_by_user_id. The row stays; session.delete() is never called; no S3 DeleteObject is ever issued (the test asserts the object_key is byte-identical before and after). This is the only deletion the product offers, because ADR-126 makes raw/ objects WORM — hard erase is physically unavailable until retain_until passes, and even then is an ops-level concern, not an app route. A paired check constraint (availability = 'deleted') = (deleted_at IS NOT NULL) makes the tombstone timestamp move in lockstep with the axis, so a “deleted” row can never lack its when, and restore must clear it.
3. Why legal hold is a hard gate, enforced in the backend
A legal hold is a court’s instruction that evidence must not disappear. So delete_document raises a typed PramaanConflictError (pramaan.document.legal_hold_active) before touching the row when legal_hold is true, and the matter-wide delete refuses the whole operation (pramaan.matter.legal_hold_active) if even one live document is held — partial deletion would be a silently broken hold. The backend is authoritative; a UI that hides the delete button is a convenience, not the guarantee. The legal_hold = false OR legal_hold_set_at IS NOT NULL constraint keeps the “who/when” of a hold honest.
4. Why the default list is live-only, with a management escape hatch
Live-only consumers (the Documents Register) must never see tombstones, so list_documents filters availability = 'live' by default, served by the partial index ix_documents_firm_matter_live_created. But the Lifecycle danger-zone rule is explicit: an archive/soft-delete concept is broken-by-construction unless a management query can return removed rows — otherwise a deleted item vanishes and can never be restored. So ?includeDeleted=true returns the tombstones for the restore/audit surface. Pickers and the register stay live-only.
5. Why “matter delete” lives here as a cascade, and what it does not touch
WORM forbids hard-dropping a matter’s S3 prefix, so a matter delete is logical: delete_matter_documents cascade-soft-deletes every live document in the matter (legal-hold-aware) and emits one matter.deleted tombstone carrying the deleted count. This is deliberately the documents-domain side of matter deletion. The matter row’s own status lifecycle (discovery/active/closed/archived) is owned by the matter domain and is intentionally left untouched here — this Story does not colonise matter-domain state; it guarantees the WORM-safe, evidence-preserving consequence of a matter being deleted.
6. Why there is no manual span here
pramaan-docs ADR-124 requires every new boundary to be instrumented or explicitly exemption-documented. The closed ManualSpanName registry has no name for a document/matter retention transition, and inventing one would be a registry change governed by the ADR, not a functions Story. These transitions are firm-scoped UPDATEs behind a route, so they ride the auto-instrumentation floor (T1): the route’s auto HTTP span + the DB span cover them, and the durable business record is the audit row each transition stamps. The exemption is written into the service docstrings as a deliberate choice, not an omission; span attributes carry only metadata, never document bytes.
Acceptance Evidence
| Acceptance criterion | Test / check |
|---|---|
| Soft-delete (no hard erase by default) | test_soft_delete_hides_then_restore_brings_back_and_audits: delete flips availability, removes it from the live list, keeps it in ?includeDeleted=true; the S3 object_key is unchanged before/after (no erase); restore returns it to live |
| Legal-hold blocks deletion | test_legal_hold_blocks_delete_until_lifted: delete returns 409 pramaan.document.legal_hold_active while held; succeeds once lifted |
| Matter-delete logical | test_logical_matter_delete_cascades_and_legal_hold_blocks_it: cascade soft-deletes all live docs (deletedCount=3); a held doc blocks the whole op (409, nothing deleted); second call is a no-op |
| Audit on delete | each test asserts the exact action sequence in ops.audit_log (document.deleted, document.restored, document.legal_hold_placed/lifted, matter.deleted); verbs resolve in the registry; routes are in the coverage manifest |
| Test | new integration suite, all flows through the real router against an isolated RLS DB; existing document + RLS-confidentiality suites stay green |
| Migration + no drift | 0110 applies forward and backward; full downgrade base → upgrade head round-trips; alembic check reports no autogenerate drift |
| Contract synced | make openapi regenerated; openapi/app-api.json advertises the five routes and the new DTOs; admin snapshot untouched |
| Tenancy | columns live on the already-RLS’d, firm-leading documents.documents; no new firm table; allowlist unchanged because the table was already listed |
| Walkthrough doc included | this document |
Document + RLS + audit suites green against an isolated Docker test DB (48 passed across retention, documents, blocks, filing-link, saved-views, status-events, RLS isolation, and firm-audit); ruff clean on changed files; make openapi idempotent; migration round-trips with no drift.
One-sentence recall. Deleting evidence is sealing an evidence locker, not shredding it: deletion flips an availability axis (row and WORM bytes retained), a legal hold hard-blocks it in the backend, and a matter delete cascades as logical soft-deletes — because ADR-126 says raw/ can never be hard-dropped.