(:Section) nodes +
RESPONDS_TO alignment (D7), the sections read route, the D33 record→strategy
teardown cascade, and a structure-aware insight brief (v2).
| Consumer | Where | What changed |
|---|---|---|
| 1 · Embed worker | src/shared/context/chunker.py::chunk_sectionssrc/workers/context_embed/processor.pymigration 0171_chunks_section_reshape |
chunk_pages is DELETED (R1), replaced by chunk_sections: blocks are
assigned to their innermost enclosing section by lexicographic (page, block)
containment; packing never crosses a section boundary (within-section split only for oversized
sections). Each section with a summary emits ONE granularity='abstract'
chunk (D15). context.chunks gains section_id NOT NULL (composite
(firm_id, section_id) FK, ON DELETE CASCADE) + granularity;
chunk_hash absorbs both (S4/F9) and the unique key widens to
(firm_id, document_id, chunk_hash, granularity); the single HNSW index becomes TWO
partial indexes, one per granularity (S3). Destructive reshape per the D12 greenfield ruling:
DELETE FROM context.chunks up front, honest truncating downgrade. The retrieval SQL
pins both legs to granularity='text' (abstract recall is B4's consumer).
Identity assert (DoD): a section-less document raises SectionsNotReadyError → SQS batch-item failure → redelivery after
segmentation lands (or honest DLQ). NEVER a silent page-window fallback.
Stale rows: re-segmentation replaces document_sections rows
in place, and the composite-FK CASCADE takes the superseded chunks with them — the S2
stale-row story, solved structurally. |
| 2 · Graph | src/shared/graph/sections.py (new)src/shared/graph/registry.py, EdgeType.RESPONDS_TOfan-out step fanout.py::_project_sections |
All graph work stays inside the graph package (F10; the CI grep gate rejects raw traversals
elsewhere). New closed-registry entries: NodeLabel.Section (key
section_id; props document_id/kind_id/kind_name/
order_index — never content) and RESPONDS_TO (Section→Section,
basis only). Projection runs as a fan-out sub-step AFTER segmentation commits and
BEFORE the enrich kick (the §5 write order), firm+matter-scoped at WRITE time via
ScopedGraph (Neptune has no RLS). RESPONDS_TO edges pair sections of the
SAME kind across documents on OPPOSING declared sides in the matter — the SOC prayer ↔
SOD prayer substrate (SS-04). Strategy documents are refused at the projection gate (SS-19).
The in-memory seam now mirrors DETACH DELETE (edges die with their node) so teardown
tests are honest. |
| 3 · Read route | GET /app/matters/{matter_id}/documents/{doc_id}/sectionsDocumentService.list_sections, DTOs in
src/domains/document/dtos/document.py |
The ordered D3 tree: id, kind key+label (counsel's words), heading path, page/block ranges,
summary, source, is_fallback, nested children. ONE joined query
(sections × kinds) filtered to the document's CURRENT extraction_run_id
(invariant 4 — a superseded run's boundaries are never served); tree assembled in memory
(DR-1/DR-2, no per-section lookups). Strategy documents ARE served — display is the one
allowlisted strategy consumer. OpenAPI snapshots committed
(appListMatterDocumentSections). The app proxy is B4's. |
| 4 · D33 cascade | DocumentService.set_home →
_teardown_machine_understanding |
Record→strategy reclassification hard-deletes the document's machine understanding in the
reclassify transaction: ALL chunks (text + abstract; the 0171 migration grants the app role
DELETE for exactly this) and the (:Section) nodes/edges (graph leg isolated —
a graph fault never fails the reclassify; the reconciler shape heals). Section ROWS stay,
display-only. Strategy→record does not rebuild here — counsel's
“Re-read against the matter” (D30) is the recovery door. |
| 5 · Insight brief | src/shared/context/insight_brief.py,
prompt document_insight_brief.v2 |
The motivating symptom closed: the seat now loads the section layer (one joined query) and
feeds the model the document as NAMED PARTS — == SECTION == headers carrying
kind + heading + summary, anchored blocks grouped beneath, orphan blocks honestly labelled
(unsectioned). Verified this_document pins now carry
section_id, DERIVED by range containment from the byte-verified quote-span (the D10
shape — the span stays the durable anchor; same innermost_section rule as the
chunker). prompt_version bumped to v2; D14 pinning + provenance unchanged; the
V-CITE byte-gate untouched. |
home='record' gate runs before ANY chunk exists, so abstract summary
embeddings inherit it by construction (integration test seeds a strategy document WITH sections and
proves zero rows, zero Bedrock spend). Graph: the projection gate refuses non-record documents before
any write. Route: serves display only. Insight: unchanged upstream gates. Neptune tenancy: every
write rides ScopedGraph's bound firm+matter — the cross-firm test reads nothing.
chunk_hash = sha256(section_id, granularity, envelope, text);
the insert is ON CONFLICT (firm_id, document_id, chunk_hash, granularity) DO NOTHING.
A re-run embeds nothing and inserts nothing. The S4 collision (a one-block section whose summary
equals its text) is proven to keep BOTH rows — unit (hash inequality) and integration (both
rows land).
| Claim | Test |
|---|---|
| Section-less record doc defers, never page-windows; embeds after sections land | tests/integration/test_context_embed_worker.py::test_embed_worker_defers_sectionless_document_never_page_windows |
| Chunks carry section identity + granularity; abstracts exist; S4 hash survives; SS-17 zero-row strategy proof | test_context_embed_worker.py::test_embed_worker_first_run_rechunk_and_exclusion +
tests/unit/test_context_chunker.py (15 tests incl. both S4 cases, boundary-never-crossed, innermost-wins) |
| (:Section) nodes + RESPONDS_TO for a two-sided matter; cross-firm reads nothing; teardown removes nodes AND edges; registry fails closed | tests/unit/graph/test_sections.py (SS-04, SS-19, D33) |
| Strategy sections never projected; opposing-side counterpart pairing; undeclared side pairs nothing | tests/unit/test_context_fanout.py (3 new projection tests) |
| Sections route serves the ordered tree, current run only, nested children, honest empty state | tests/integration/test_document_sections_route.py |
| Reclassify record→strategy deletes chunks (all granularities) + graph teardown, keeps display sections | tests/integration/test_document_home.py::test_reclassify_to_strategy_tears_down_chunks_keeps_sections |
| Insight brief reads structure; pins carry derived section_id; matter/general_law pins never do; orphan blocks visible | tests/unit/context/test_insight_brief.py (4 new tests) |
| Migration shape pinned: destructive-first, composite FK, granularity in CHECK/unique/two partial HNSW, DELETE grant, honest downgrade | tests/unit/test_chunks_section_reshape_migration.py + CI Alembic round-trip (run locally: up/down/up clean, alembic check no drift) |
| # | Answer |
|---|---|
| 1 | New cloud capability: none. Neptune writes ride the existing graph package/connection; DB changes are one migration on the existing cluster; no new queue/IAM/network path. |
| 2 | No new grant needed — the only privilege change is SQL-level (GRANT DELETE ON context.chunks TO pramaan_app) inside migration 0171 itself. |
| 3 | Checks that go red: the migration-shape guard test, the defer/S4/SS-17 embed tests, the SS-19/SS-04 graph tests, the OpenAPI drift check, CI Alembic round-trip. |
| 4 | Red observed: the sections-route test failed red twice during development (missing S3 stub; asyncpg text[] literal) before green; the defer test fails red by construction if the assert is removed (page-window fallback would return EMBEDDED, not raise). |
| 5 | Legacy deleted IN THIS PR: chunk_pages (function + all callers + its tests), the single-granularity chunk shape (uq_chunks_firm_document_hash, single HNSW index). Nothing stays temporary. |
| 6 | Deploy smoke: the embed worker's changed path is exercised by the DB-backed integration tests in CI; worker deploy canary remains UNENFORCED U2 (pre-existing, not widened here). |