Type a query, see which strategy each router picks and which product comes back.
This replaces the notebook's compare_query table with something you can show in a browser.
Build brief: docs/2026-09-09-home-depot-route-demo-design.md.
cp .env.example .env.local # then fill in the values
npm install
npm run dev # http://localhost:3000node scripts/smoke.mjs checks the Qdrant path (dense + server-side bm25) before anything else.
node scripts/check.mjs is the logic self-check (TM2C2 fusion, NDCG, and LLM-based bucketing).
node scripts/title.mjs backfills display titles for the 69,761 points (56.1%) whose
product_title was missing from train.csv/test.csv. Payload only, so retrieval and the
bakeoff numbers are untouched. Check the model against the titles that do exist first:
node scripts/title.mjs --selfcheck # parser + scorer asserts, no network
node scripts/title.mjs --limit 40 --dry --test # scored against real titles as gold
node scripts/title.mjs # the full run, ~$1Needs OPENROUTER_API_KEY. Generated titles carry title_generated: true in the payload.
Six arms, all from one /api/search response. The three that make a decision:
| Mode | Decision comes from |
|---|---|
| Hybrid routing (default) | our encoder-router /classify → dense_only / sparse_only / pure_rrf, with the p_dense/p_sparse hedge margin |
| Fusion RRF | TM2C2 score fusion at the weight that same router returns from /classify/weighted: α·sparse + (1−α)·dense over top-normalized leg scores. A router without that endpoint falls back to the unanchored p_sparse/(p_sparse+p_dense) |
| LLM based | hosted LLM classifier score 0–9, bucketed (≤2 dense, ≤6 hybrid, else sparse). Greys out to "offline" if unreachable |
…plus three that don't: fixed_rrf (the "replace this" baseline) and the raw dense_only and
sparse_only legs. All six are selectable on their own, and Compare all fans them into six
columns at once. Green dot = same top-1 as hybrid routing, red = diverged. The divergence is the
eye-test, and each column carries its own NDCG@N.
One query = 3 Qdrant round-trips: dense(20), sparse(20), native RRF(20). FETCH_LIMIT = 20 matches
the notebook's bakeoff numbers. Fusion RRF reuses the dense+sparse lists — no extra Qdrant call,
but one more router call for the weight.
All keys stay server-side; the browser only ever calls /api/search and /api/image.
Home Depot's own site search is bot-walled (403), so tier 1 of the two-tier plan is dropped —
image search is the only tier. Provider is Brave (BRAVE_API_SEARCH); results are filtered
site:homedepot.com so cards get real HD product shots. Google's Custom Search JSON API is not an
option (closed to new customers, shuts down 2027-01-01). Without a key every card renders the
letter placeholder and the demo still works.
Cost is bounded by unique products ever displayed, not by queries. Three guards, all covered by
node scripts/check-cache.mjs (which runs against a fake endpoint, so the test itself costs $0):
- Persistent cache —
product_uid → urlin gitignoreddata/images.json. Survives dev restarts. - Known-misses are cached too — a product Brave has no image for stores
null, so it is never re-fetched. - In-flight dedupe — the compare view mounts the same product in several columns at once. All those cards share one promise, so they cost one call between them, not six.
A rate-limit or outage is deliberately not cached: it would otherwise poison the cache with a permanent miss for a product that does have an image. Those uids stay absent and retry on the next render. Calls are also queued one at a time (~1/sec) to stay inside Brave's free-plan rate limit.
Measured: 4.6 new lookups per query across 15 varied queries. At Brave's $5/1k that is $0.023 per query, and the $5 monthly credit (~1,000 requests) covers roughly 200 fresh queries a month. A 20-query rehearsed demo costs ~92 lookups once, then $0 on every re-run.
The same document scores differently in every column because a different function produced each number, so each card labels its raw score with the metric behind it:
| column | route | metric | raw (doc 168889) |
|---|---|---|---|
| Hybrid routing | dense_only |
cosine | 0.692 |
| Fusion RRF | — fuses both lists | TM2C2 | 0.955 |
| LLM based | pure_rrf |
RRF | 0.625 |
| Fixed RRF | pure_rrf |
RRF | 0.625 |
| Sparse only | sparse_only |
BM25 | 27.696 |
An RRF score is a sum of reciprocal ranks (Σ 1/(60+rank)), so it is structurally small and can
never resemble a cosine; a TM2C2 score is a convex blend of two top-normalized legs, so it lives in
[0,1] and 1.0 means "top of both". Same document, four scoring functions, four numbers.
Raw scores are therefore not comparable across arms — dense cosine is [-1,1], BM25 is unbounded and routinely above 25, RRF sums sit near 0.02. BM25 and cosine differ in both scale and offset, so each card also shows a z-score: how many standard deviations that hit stands above the mean of its own arm's fetched top-20. That is invariant to scale and offset, so it reads across arms:
sparse_only raw 27.696 -> +3.08σ
dense_only raw 0.692 -> +3.12σ
Deliberately not ratio-to-max: that pins every rank-1 hit to exactly 1.00, so the top-1 compare
view would show 1.00 in all six columns and say nothing. The z-score still separates them
(hybrid +3.12σ vs the fusion arm's +2.42σ on the same query), which is the "how confident is this arm"
signal the eye-test wants.
Display only — RRF fuses on rank, so normalisation never touches the ranking. The pure functions
live in lib/fusion.ts with no Qdrant import, and scripts/check.mjs imports those exact
functions rather than re-implementing them.
The grades are Kaggle's own — three raters per pair, averaged to a float 1.00–3.00, shipped as
train.csv in the "Home Depot Product Search Relevance" set. data/qrels.json is that file keyed
by search term (74,062 judgments over 11,793 terms, mean 2.38); scripts/qrels.py derives it and
cross-checks against the dataset repo's qrels.parquet. Anything under 2.0 counts as zero, the
same REL_THRESHOLD the notebook's bakeoff binarizes at.
Gold exists only for Home Depot's own ~11.8k search_terms, so a query you type yourself usually
has none and every arm shows — rather than 0.00. Measured over a 45-term sample, 62% of graded
terms put at least one graded product in some arm's top 10; the rest score zero across the board,
which is the shallow-pool effect below, not six broken arms.
The queries below were picked by measuring them against the live index, because this is the
comparison the demo exists to make. These numbers were measured before the TM2C2 swap, so the
fusion column is the old weighted-RRF arm and has not been re-run; the other five arms are
unchanged by that commit. The set that actually ships in the UI is the prepared-query menu,
regenerated with python scripts/prepared.py and carrying its own measured notes.
| query | dense | sparse | fixed RRF | weighted RRF (superseded) | hybrid | judged coverage |
|---|---|---|---|---|---|---|
lazy susan rev a shelf spacer |
0.759 | 0.933 | 0.821 | 0.829 | 0.933 | 7–10 of 10 |
stone effects backsplash cool fushion |
0.380 | 0.785 | 0.530 | 0.712 | 0.530 | 7–9 of 10 |
3/16 x 12 x 165 bubble cushion |
0.878 | 0.618 | 0.833 | 0.827 | 0.618 | 7–9 of 10 |
wood garage door |
0.000 | 0.640 | 0.092 | 0.000 | 0.640 | 0–7 of 10 |
The first three were found by scoring all 654 terms carrying 12 or more human grades against the live index, then keeping those where every arm's top 10 is at least 70% judged. They need no LLM judge and carry no unjudged-hole caveat — these are the ones to lean on when the metric is challenged:
lazy susan rev a shelf spacer— hybrid routing 0.933 against fixed RRF 0.821, with all ten ranks of the winning arm human-graded and all 12 judgments relevant. The router beating the baseline it replaces, on nothing but Kaggle's raters.stone effects backsplash cool fushion— the widest separation in the corpus (0.41). Sparse holds 0.785; plain RRF buries it to 0.530; weighted RRF recovers 0.712. The router hedged to RRF and lost 0.255, which is why it stays in.3/16 x 12 x 165 bubble cushion— dense 0.878 against sparse 0.618. Proof the graded pool does not simply favour BM25, and a second case the router gets wrong.
wood garage door is the original fusion-buries-relevant case and the one with real holes — sparse's
0.640 rests on 7 judged docs while dense's 0.000 rests on none, so read it as rank movement, not as
dense failing. A query with no gold at all shows what — looks like.
ndcgAt mirrors the bakeoff's ranx call: linear gain (a 3.0 counts 1.5x a 2.0), log2 rank
discount, IDCG taken from the judged relevant set. /api/search returns the grades, not a score —
the top@N slider re-scores client-side with no refetch, as does the gold-source switch. Kaggle
judged ~6 products per term, so scripts/judge_qrels.py grades the retrieved pool itself onto the
same scale; a human grade always wins and the mode bar names which provenance produced the number.
Read them as a within-query comparison, not as published NDCG, and do not read them as "which arm is better". Measured over 60 graded queries — again before the TM2C2 swap — 79-89% of every arm's top 10 was never judged and therefore scores zero:
| arm | judged docs in its top 10 | unjudged |
|---|---|---|
| sparse only | 2.08 | 79% |
| hybrid routing | 1.97 | 80% |
| fixed RRF | 1.78 | 82% |
| weighted RRF (superseded) | 1.58 | 84% |
| dense only | 1.15 | 89% |
Dense is hit hardest, and not by accident. Kaggle's raters graded pairs that Home Depot's own 2016
production search surfaced, so the judged pool is lexically sourced and BM25 was always going to
find more of it. The same rows show judged ≈ relevant (dense: 1.15 judged, 1.13 relevant) — the
qrels are a list of known-good products, not a labelled sample of the catalog.
wood garage door is the case to have in mind. Dense returns a Clopay 16 ft x 7 ft Ultra-Grain
Walnut garage door (181986) that no rater ever saw, so it scores 0.000; sparse returns two Clopay
4 in. x 3 in. wood samples (163499, 142676) graded 3.0, so it scores 0.640.
What survives that caveat is the claim the demo is actually making: fusion pushed the graded documents down. That is a statement about where known-relevant docs moved between two rankings, and it holds whatever the pool missed. "Sparse beats dense" does not hold, and is not claimed.
The dataset repo also carries minted_qrels.parquet, and it is the wrong tool here. Those 1,296
queries were written by a model from a corpus document, so the gold is that seed document by
construction rather than a rater's judgment — one doc per query, relevance flat at 2.0. Three
consequences: NDCG collapses to 1/log2(rank+1) and carries nothing the doc's rank doesn't; a
second product that answers just as well scores zero, and this catalog has literal duplicates
(135485 is the same MOEN rod as 100636); and ~88% of a minted query's tokens already occur in
its gold doc, which favours BM25 — visible in the bakeoff's own means, sparse_only 0.4269 vs
dense_only 0.3432. Most decisive for this demo: with one gold doc per query it cannot show
fusion pushing several graded products down, which is the whole point of the table above.
- grade badges on cards — the per-card relevance badge from the brief. The grades are loaded and NDCG is scored per arm; what's missing is the per-product badge and the hit@1 marker.