Knowledge: verifiable citations — quote checking, then chunk spans

[joeyorlando]

Problem

Search results cite a document: title, source URL, connector, and the chunk's position. A reader can open the source but not see which sentence supports the claim. With context expansion a returned passage can span several chunks, so "which part of this did the answer use" is a larger question than it used to be.

Example

An agent answers that the data retention period is 90 days, citing a 60-page policy document. To verify, someone opens the document and searches it by hand. If the agent had the number wrong, nothing in the citation reveals that — the document is genuinely relevant and genuinely cited.

Where

• platform/backend/src/knowledge-base/chunker.ts — where offsets would be recorded
• platform/backend/src/knowledge-base/query.ts — ChunkResult.citation
• platform/backend/src/archestra-mcp-server/ — the tool result the model sees

Approach

Do the cheap version first; it delivers most of the value and needs no schema change.

Quote verification. Ask the answering model to include a verbatim quote with each claim, then check the quote actually appears in the cited chunk. A quote that does not appear is a fabrication, and it is caught programmatically rather than by a reader. This is a prompt plus a string match, and it doubles as a hallucination check.

Stored offsets. Record character offsets when chunking and surface them in the citation. Two limitations to weigh before building on them:

• Offsets point into extracted plain text, not into the rendered source. There is no way to deep-link a highlight into a Confluence page or a Jira description, so the payoff is confined to a document viewer inside Archestra, which would also have to be built. PDFs would want page and bounding box instead, a different data model.
• Offsets shift when a document is re-ingested, so citations stored against an older version silently point at the wrong span. Chunk-level citations degrade more gracefully.

Done when

• An answer's claims can be verified against the cited source without a reader opening the document and searching it manually.
• A claim whose quote does not appear in the cited chunk is detectable.

Notes

Ranked below the retrieval items deliberately. This is a trust and audit feature rather than a retrieval quality one, and the expensive half has a narrow payoff.

[joeyorlando]

A follow-up look at the render path found that the cheap half of this is cheaper than the issue assumes — most of it is deletion, not new plumbing.

The matched passage is already in hand at render time, and we throw it away.

backend/src/knowledge-base/query.ts:26-38 — ChunkResult carries the passage itself:

interface ChunkResult {
  content: string;        // the matched passage
  score: number;
  chunkIndex: number;
  metadata: Record<string, unknown> | null;
  citation: { title; sourceUrl; documentId; sourceId; connectorType };
}

These are returned verbatim to the model by archestra-mcp-server/knowledge-management.ts:673-678. But the UI drops everything except citation, and then keeps only one chunk per document — frontend/src/components/chat/knowledge-graph-citations.tsx, extractCitations:

const seen = new Set<string>();
…
for (const chunk of results) {
  const c = chunk.citation;
  if (!c?.documentId || seen.has(c.documentId)) continue;   // ← every later match discarded
  seen.add(c.documentId);
  citations.push({ title, sourceUrl, connectorType, documentId, sourceId });
}

Note the loop variable is chunk and chunk.content is in scope on that line. ExtractedCitation has no text field at all, so the passage never reaches the chip. The chip then links to the whole document, and renders as an inert <div> when there is no sourceUrl.

So "show the quote" needs no schema change and no re-ingest: stop de-duplicating to one chip per document, and render chunk.content on the chip. The string-match quote check described above sits naturally on top and doubles as the hallucination guard.

This does not change the assessment of stored offsets — both objections in the issue body still hold, and the positional data genuinely is not there. Confirmed absent from kb_chunks: pageNumber, boundingBox, charOffset, startOffset — zero matches across backend/src/knowledge-base and shared. The PDF extractor discards page structure entirely.

One trap for any later indexOf-into-document approach: chunk content is not a clean substring of its parent document. chunker.ts:94 prepends `TITLE: ${title}\n\n` to every chunk and applies trimStart() to the body, so a naive offset search against the stored document text will miss.

[piercypixel]

Completed by #7199, merged as 2543225.

[eph5xx]

@piercypixel Flagging that closing this as completed may be premature. #7199 shipped Phase 1, but the issue scoped more than that. The parts:

Part 1: quote verification (Phase 1). Done in #7199.
The model backs each claim with a verbatim quote tagged with a derived ref (documentId#chunkIndex), and knowledge-base/quote-verification.ts checks each quote against the chunks the tool returned that turn. The results (matched, wrong_ref, failed, unverifiable) feed a rag_quote_verification_total counter. It's log-only, built-in chat only, and needs no schema change. This covers both "Done when" bullets.

Part 2: show the quote on the citation chip. Not done.
This is the cheap half @joeyorlando described above ("stop de-duplicating to one chip per document, and render chunk.content on the chip"). #7199 never touched frontend/src/components/chat/knowledge-graph-citations.tsx. It still de-dupes to one chip per document (:82), and ExtractedCitation has no text field (:25-31), so chunk.content never reaches the chip. The reader sees quotes in the model's Sources section, but the chips still link to the whole document. Small change, mostly deletion.

Part 3: stored offsets / chunk spans (Phase 2). Not done.
Never built. The "out of scope, ships separately" line was #7199's own PR scoping, not a decision recorded on this issue. Both objections in the body still hold, and the positional data is still absent: no startOffset, endOffset, pageNumber, or boundingBox on kb_chunks (backend/src/database/schemas/kb-chunk.ts). This one is a larger effort for a narrow payoff, so it deserves an explicit keep-or-WONTFIX call instead of being dropped silently.

So Part 1 is done, Parts 2 and 3 aren't. I'd suggest reopening, or filing a quick follow-up for Part 2, plus a decision on Part 3.