context.chunks + structure-aware chunker + embed workerdocument.md (the synthesized render) by character count, every hit
would be a floating window with no honest coordinate back into the record — and the
Context Engine's whole "Why + Cite" acceptance test collapses the first time a court
asks where a retrieved passage lives. And if the model id / dimension /
region that produce those vectors were baked into code, swapping the embedder would
become a code-change-plus-deploy instead of the one-config-line the RFD promises, and
a silent dimension mismatch would corrupt the index. CE-3 builds the semantic leg of
L2 the right way once: chunks born from document_pages.blocks with a
byte-verifiable span envelope, embedded by a config-pinned model, idempotent on
chunk_hash so a re-run never pays Bedrock twice.
A home='record' document's OCR pages are chunked deterministically into
context.chunks (pgvector embedding + generated fts,
each carrying its page/block span envelope), embedded by an SQS worker using
cohere.embed-v4 pinned in config against the pinned us-east-1 Bedrock
region, idempotent on chunk_hash so a deterministic re-chunk is a no-op.
Think of a conveyor belt (the enrich kick lands on the queue), a ruler (the chunker
packs OCR blocks into ~450-token chunks, never crossing ~800, overlapping neighbours by
one block, and slicing a single oversized block at sentence boundaries), and a stamp
(the embedder turns each chunk's text into a fixed-dimension vector). Every chunk keeps
a note of exactly which pages and blocks it came from, so a later retrieval hit resolves
straight back to real page bytes. The stamp reads its ink colour (model id), its stamp
size (dimension), and which post office it uses (region) from a config card — never from
the machine's wiring. And each chunk is fingerprinted (chunk_hash): if the
belt runs the same document again, the fingerprints already on file mean nothing gets
re-stamped.
| Surface | Change |
|---|---|
Migration 0157_context_chunks_pgvector |
CREATE EXTENSION IF NOT EXISTS vector (idempotent, mirrors the
pg_trgm precedent in 0132) + context.chunks: firm/matter/document ids,
chunk_index, the (page_start/end, block_start/end) span
envelope, text, token_count, chunk_hash
(sha256, the idempotent re-embed key), embedding vector(1024) (HNSW
cosine index), a GENERATED fts tsvector ('simple' config, GIN index),
and embedding_model. RLS cloned from document_pages;
SELECT/INSERT/UPDATE, never DELETE. Unique on
(firm_id, document_id, chunk_hash) — the no-op key. Up/down/up + base
roundtrip + alembic check clean. |
src/shared/context/chunker.py (new) |
Deterministic, structure-aware chunker: greedy block packing to
target_tokens, hard max_tokens ceiling,
overlap_blocks overlap, sentence/word split for a single oversized
block. Reads document_pages.blocks (never document.md),
preserves original block indices, admits only succeeded pages, and
produces byte-identical output for the same input — the idempotency guarantee the
worker relies on. Sizes come from ChunkConfig.from_settings. |
src/shared/context/embedder.py (new) |
BedrockCohereEmbedder.from_settings: model id, region, and
dimension are all constructor arguments sourced from settings — never code. Batches
over Cohere's 96-text limit, accepts both v3/v4 response shapes, and
fails closed on any vector whose length ≠ the configured dimension
(a wrong-dimension embedding is never returned, let alone stored). boto3 is lazy so
tests and the offline path never touch it. |
src/workers/context_embed/ (new) |
SQS worker: envelope.py (the ids-only enrich-kick pointer +
trace context), processor.py (process_embed), and
handler.py (batch/DLQ, mirrors the extraction worker). The processor
sets firm RLS, loads the document, gates on home='record'
(strategy is structurally excluded), chunks the pages, prefetches stored hashes in
ONE query (DR-2), embeds only the new chunks, and inserts them in ONE statement
(DR-5) with ON CONFLICT DO NOTHING. Query count per record is bounded
and constant (DR-1). |
src/shared/settings.py |
Config, not code (pramaan-docs ADR-151): CONTEXT_EMBEDDING_MODEL_ID
(cohere.embed-v4), CONTEXT_EMBEDDING_DIMENSION (1024, must
equal the vector(D) column), CONTEXT_BEDROCK_REGION
(us-east-1), and the chunk sizes
(CONTEXT_CHUNK_TARGET_TOKENS/MAX_TOKENS/OVERLAP_BLOCKS). |
manual_span.py + CI images |
Two registered CONSUMER spans (CONTEXT_EMBED_BATCH,
CONTEXT_EMBED_PROCESS_RECORD); the Bedrock invoke_model is
botocore-auto-instrumented (T1). CI/test-db Postgres services swap to
pgvector/pgvector:pg16/pg17 so the extension binaries exist
(staging RDS already permits CREATE EXTENSION, proven by 0132 pg_trgm). |
Swapping the default embedder is one config line (CONTEXT_EMBEDDING_MODEL_ID)
plus a batch re-embed of rows where embedding_model != current —
chunk_hash keeps it idempotent and resumable. vector(D) is
fixed-dimension, so a successor whose dimension differs is NOT a plain re-embed: it rides
the documented dimension-migration path (new column + new HNSW index + dual-write +
cutover + drop). The config dimension and the migration column literal are pinned to move
together, by construction.
tests/integration/test_context_embed_worker.py drives
process_embed against a REAL Postgres (with pgvector) as the runtime
pramaan_app role, from the empty first-run state: first-run chunks land for
a home='record' document (5 rows, each a real vector(1024)
stamped cohere.embed-v4); a deterministic re-chunk is a no-op
(NO_NEW_CHUNKS, zero embedded, row count unchanged — no double Bedrock
spend); a home='strategy' document is skipped
(SKIPPED_NOT_RECORD, nothing embedded). Hermetic unit gates:
test_context_chunker.py (determinism, overlap, oversized split, structural
filters), test_context_embedder.py (config-read, both response shapes,
batching, dimension fail-closed), test_context_embed_handler.py
(DLQ/partial-failure: malformed acked-and-dropped, transient fault →
batch-item failure, one bad record doesn't sink a good one). Ran locally: unit suites +
integration green; migration base roundtrip + alembic check no drift;
ruff clean on changed files; make openapi no diff (no route/DTO
change).
context.chunks — the
semantic half of retrieval, built once on pgvector, where a re-run costs nothing.