Problem
There is no way to tell whether a retrieval change helps. Chunk size, context expansion radius, contextual retrieval, hybrid on/off, embedding model, and reranker choice are all tunable, and every one of them is currently tuned by argument rather than measurement.
This blocks every other item on this roadmap. A BM25 ranker, a second chunk granularity, or per-chunk contextual retrieval each cost real work, and none of them can be shown to have paid off.
Example
An operator raises the context expansion radius from 1 to 3 because answers seemed to be missing surrounding detail. Answers get longer. Nobody can say whether they got more correct, and the extra tokens are now on every query forever. Six months later a different operator lowers it back for the same quality of reason.
Where
platform/backend/src/knowledge-base/query.ts — queryService.query is the entry point to measureplatform/backend/src/config.ts — the kb block holds the knobs under test- Fixtures in the style of
platform/backend/src/knowledge-base/query.test.ts
Approach
- A golden set. Query, plus the document ids that should be retrieved for it. Store it as a fixture so it lives in the repo and reviews like code. Seed it from real questions where the expected source is known.
- Scoring. Recall@k and MRR at minimum. Recall@k answers "did we retrieve the right document at all", which is the failure that matters — a reranker cannot fix a document that never entered the candidate set.
- A runner. A script that runs the golden set against the current configuration and prints the scores.
- Comparison. Run the set under two configurations and print the delta per query, not just the aggregate. Aggregates hide the case where a change fixes five queries and breaks five others.
Keep it offline and runnable locally against a seeded corpus. This does not need to be a product surface to be useful; a script a developer runs before and after a change is most of the value.
Done when
- A developer can run one command, change one setting, run it again, and see which queries improved and which regressed.
- The golden set is a checked-in fixture, extendable by adding a row.
- Scores are reproducible across runs given the same corpus and configuration.
Notes
Retrieval scoring needs a fixed corpus, so the runner should seed its own rather than depend on whatever a developer happens to have synced.
Problem
There is no way to tell whether a retrieval change helps. Chunk size, context expansion radius, contextual retrieval, hybrid on/off, embedding model, and reranker choice are all tunable, and every one of them is currently tuned by argument rather than measurement.
This blocks every other item on this roadmap. A BM25 ranker, a second chunk granularity, or per-chunk contextual retrieval each cost real work, and none of them can be shown to have paid off.
Example
An operator raises the context expansion radius from 1 to 3 because answers seemed to be missing surrounding detail. Answers get longer. Nobody can say whether they got more correct, and the extra tokens are now on every query forever. Six months later a different operator lowers it back for the same quality of reason.
Where
platform/backend/src/knowledge-base/query.ts—queryService.queryis the entry point to measureplatform/backend/src/config.ts— thekbblock holds the knobs under testplatform/backend/src/knowledge-base/query.test.tsApproach
Keep it offline and runnable locally against a seeded corpus. This does not need to be a product surface to be useful; a script a developer runs before and after a change is most of the value.
Done when
Notes
Retrieval scoring needs a fixed corpus, so the runner should seed its own rather than depend on whatever a developer happens to have synced.