Problem
Contextual retrieval currently generates one document-level context per document and indexes it with every chunk of that document. That works well for a document about one thing, and poorly for a long or heterogeneous one, where a single summary describes the whole and situates no particular chunk.
Example
A quarterly engineering review covers six projects across forty pages. The document-level summary says it is a quarterly review covering those six projects. A chunk deep inside the third project's section, discussing a migration rollback, gets a summary that names all six projects and does not make clear that this passage is about the third. A question naming that project retrieves the section header, not the rollback discussion.
Where
platform/backend/src/knowledge-base/contextual-retrieval.ts — buildDocumentContext, one call per documentplatform/backend/src/knowledge-base/connector-sync.ts — chunkAndStore, where the header is applied to every chunkplatform/backend/src/database/schemas/kb-chunk.ts — contextual_header, already per chunk in storage
The storage is already per chunk. Only the generation is per document.
Approach
Generate a short passage per chunk that situates that chunk within its document, and store it in the existing contextual_header column.
The blocker is cost, not design. A naive implementation is one LLM call per chunk, roughly 50 to 100 times the ingest cost of the document-level pass. Before this is viable:
- Prompt caching. The document body is identical across every chunk of that document, so it should be cached rather than resent per call.
- Batching. Several chunks per call, or a provider batch API where available.
- Selectivity. Long or heterogeneous documents benefit; a two-paragraph document does not. A length or structure threshold may capture most of the gain at a fraction of the cost.
Done when
- Per-chunk context is generated and indexed, behind its own setting, with the document-level pass still available as the cheaper option.
- Measured ingest cost per document is recorded next to the retrieval quality delta from the evaluation harness.
Notes
Only worth doing if it beats the document-level pass by enough to justify the cost. That comparison needs the evaluation harness.
Problem
Contextual retrieval currently generates one document-level context per document and indexes it with every chunk of that document. That works well for a document about one thing, and poorly for a long or heterogeneous one, where a single summary describes the whole and situates no particular chunk.
Example
A quarterly engineering review covers six projects across forty pages. The document-level summary says it is a quarterly review covering those six projects. A chunk deep inside the third project's section, discussing a migration rollback, gets a summary that names all six projects and does not make clear that this passage is about the third. A question naming that project retrieves the section header, not the rollback discussion.
Where
platform/backend/src/knowledge-base/contextual-retrieval.ts—buildDocumentContext, one call per documentplatform/backend/src/knowledge-base/connector-sync.ts—chunkAndStore, where the header is applied to every chunkplatform/backend/src/database/schemas/kb-chunk.ts—contextual_header, already per chunk in storageThe storage is already per chunk. Only the generation is per document.
Approach
Generate a short passage per chunk that situates that chunk within its document, and store it in the existing
contextual_headercolumn.The blocker is cost, not design. A naive implementation is one LLM call per chunk, roughly 50 to 100 times the ingest cost of the document-level pass. Before this is viable:
Done when
Notes
Only worth doing if it beats the document-level pass by enough to justify the cost. That comparison needs the evaluation harness.