RFC: Retrieval Diagnostics API for RAG Pipelines

[rautaditya2606]

Is your feature request related to a problem?

Yes.

When a Haystack RAG pipeline returns poor or empty results, there is currently no native way to determine where the retrieval process failed. While Haystack provides useful debugging primitives such as include_outputs_from, tracing, and breakpoints, developers still have to manually inspect intermediate outputs and infer the root cause themselves.

Some common scenarios are:

1. Empty retrieval

A query returns zero documents. The developer has no indication whether this is caused by embedding mismatch, metadata filters, indexing problems, or simply the absence of relevant content.

2. Metadata filter exclusion

A query with metadata filters unexpectedly returns no documents. Haystack exposes the empty result, but developers must manually inspect the underlying document store to understand whether filters excluded otherwise relevant documents.

3. Reranker context loss

A reranker is introduced to improve retrieval quality, but answer quality decreases.

Without manually inspecting intermediate component outputs, it is difficult to determine whether:

• the retriever never found relevant documents, or
• the retriever found them but the reranker removed or significantly demphasized them.

These represent different failure modes but currently require manual investigation.

---

Proposed solution

Introduce a framework-supported retrieval diagnostics API that interprets intermediate pipeline outputs and classifies common retrieval failures.

Rather than exposing only raw retriever and reranker outputs, Haystack could provide a higher-level diagnostic object.

Example:

from haystack.diagnostics import diagnose_retrieval

result = diagnose_retrieval(
    pipeline=rag_pipeline,
    query="What is the return policy?",
    ranking_threshold=0.7,
)

print(result.failure_class)
print(result.explanation)
print(result.details)

One possible initial taxonomy could include:

Failure class	Description
NO_RESULTS	Retriever returned zero documents
FILTER_EXCLUSION	Metadata filters excluded candidate documents
SCORE_BELOW_CUTOFF	Retrieved documents scored below a configurable threshold
CONTEXT_LOSS	Documents retrieved upstream were removed or significantly reordered by downstream ranking
EMPTY_CONTEXT	Retrieved documents contain no usable textual content

This API could internally build upon existing mechanisms such as include_outputs_from rather than introducing a separate execution path.

---

Why this belongs in Haystack

Haystack already provides the primitives required for debugging pipeline execution.

This proposal focuses on adding a framework-native interpretation layer that converts intermediate pipeline outputs into actionable diagnostics without requiring users to manually inspect component outputs or build custom debugging utilities.

It complements existing observability features rather than replacing them.

---

Prior prototype

I implemented an external prototype (haystack-diagnostics) to validate this idea.

GitHub repository: https://github.com/rautaditya2606/haystack-diagnostics

During development it successfully identified multiple real-world retrieval issues, including:

• retrieval failures caused by metadata filters
• reranker-induced context loss
• empty retrieved context caused by malformed documents

The prototype uses Haystack's existing APIs (include_outputs_from) to perform single-pass inspection without requiring additional pipeline executions.

The goal of this proposal is not to upstream that project directly, but to discuss whether a small, generic retrieval diagnostics API belongs in Haystack core.

---

Related discussion

This proposal is complementary to #11286 (Standardized Pipeline Run Artifacts).

That RFC focuses on recording pipeline execution.

This proposal focuses on interpreting retrieval behavior and classifying common retrieval failure modes.

---

Alternatives considered

• OpenTelemetry tracing provides excellent observability but requires additional infrastructure and still leaves interpretation to the developer.
• include_outputs_from exposes intermediate outputs but requires manual inspection and does not provide structured diagnostics.
• External tooling works but fragments the debugging experience and requires additional installation and maintenance.

---

Scope

Initial MVP:

• diagnose_retrieval()
• structured diagnostic result
• initial retrieval failure taxonomy
• implementation built on existing Haystack execution primitives

Out of scope:

• document store validation
• CLI
• debug bundle serialization
• async support

---

I'd be happy to prototype this if the maintainers think this direction aligns with Haystack's architecture. Given the new public API surface, I'm also happy to write a design proposal before opening a PR if that's preferred.

[zeweihan]

Really strong RFC — the failure taxonomy is the right abstraction level, and framing diagnostics as an interpretation layer over include_outputs_from (rather than a new execution path) is exactly the right scoping instinct.

One gap worth surfacing from the regulated-document (legal/financial/compliance) side: every failure class in the current taxonomy is an absence failure. NO_RESULTS, FILTER_EXCLUSION, SCORE_BELOW_CUTOFF, EMPTY_CONTEXT all describe "we got nothing, or nothing usable." Those are the detectable failures — a human staring at an empty answer would notice them too. The class that actually hurts in production RAG over dense corpora is the one that looks like success: retrieval returned documents with good scores, the answer is fluent and confident — but the retrieved content is correct-yet-misattributed, or partial in a way that changes the operative meaning. That's the NONE-that-isn't case the current taxonomy would label healthy.

Concretely, two additions that would keep the diagnostics API from manufacturing false confidence:

1. Make the diagnostic object carry provenance per retrieved doc, not just score.
Today failure_class = NONE resolves to "scored above threshold." For a contract corpus that's insufficient: a chunk can score highly and still be untraceable (no source page / row / cell / section anchor), which means the answer it produced cannot be cited back to the controlling provision. The binary NO_RESULTS ↔ NONE hides the plausible-but-wrong middle. If the diagnostic record carried (score, source_page, source_section, structural_role, traceable: bool) per returned doc, then UNTRACEABLE_RESULT becomes a first-class class — "we retrieved something confident, but we can't tell you which clause it came from, so treat the answer as unverified." That single class is, in my experience, where ~all the real production risk lives in regulated RAG.

2. CONTEXT_LOSS should be weighted by what got demoted, not just that something did.
A reranker dropping a doc from rank 1 → 4 is currently visible only as a score delta. But the severity depends entirely on what was demoted: demoting the controlling late-delivery penalty clause below the cutoff, while leaving boilerplate clauses ranked above, produces an answer that's wrong about the number a dispute turns on — far more dangerous than demoting a definitional clause. A bare "CONTEXT_LOSS detected" flag is informational; pairing it with a hint about the demoted doc's structural role (or even a cheap "operative-numeric-content present in demoted doc" signal) turns it from visible into actionable. Otherwise a reviewer sees "reranker reshuffled things" and moves on.

A small one that connects to FILTER_EXCLUSION: the highest-value complement is not "filters excluded N candidates" but "the filter excluded a candidate that would have changed the answer" — i.e., surface excluded docs (with scores) alongside the returned set, so an auditor can verify the filter didn't silently drop the controlling clause rather than just boilerplate. include_outputs_from already has the data; the diagnostic just needs to present the excluded set with scores rather than confirm the filter ran.

Worth naming explicitly since the RFC says it complements #11286 (Standardized Pipeline Run Artifacts): if run artifacts record what happened and this API classifies retrieval health, the missing third leg is answer-level provenance — stamping which retrieved doc (and which span within it) produced each claim in the generated answer. The diagnostics API is the natural place to demand that stamp exist before declaring NONE, because that's the only signal that distinguishes "retrieval worked" from "retrieval worked and is defensible."

Happy to help scope the provenance fields on the diagnostic record if there's appetite — this is the piece that turns the API from a debugging convenience into something an auditor or reviewer can actually rely on.

[rautaditya2606]

Thank you @zeweihan for the thoughtful feedback—I really appreciate you taking the time to review the RFC.

I agree with your observation that the current taxonomy primarily captures absence failures, whereas "plausible but wrong" retrieval is a different class of production risk, especially for regulated document workflows.

I intentionally kept this RFC focused on diagnosing the retrieval pipeline itself rather than answer provenance or document attribution, mainly to keep the scope small enough to be implementable. I do think ideas like provenance metadata, traceability, and richer document-level diagnostics would complement this API well, but they probably deserve a dedicated proposal instead of expanding this one.

Your point about surfacing excluded documents alongside returned results is particularly interesting since that information may already be available from pipeline components and could improve auditability without fundamentally changing the design.

Thanks again for the detailed review and it's given me several ideas for future follow up work.